Method and apparatus for switching between sensors

ABSTRACT

Switching among sensor feeds for optimum performance includes disposing a processor in communication with sensors, each said sensor providing a primary data stream. A data stream standard is established in the processor. The primary data streams are communicated from the sensors to the processor, and are compared against the data stream in the processor. A secondary data stream is selected from among the primary data streams or synthesized from one or more of the primary data streams, based on which primary data stream(s) most closely match the data stream standard. The secondary data stream is communicated to a data stream recipient. The data stream recipient may identify input in the secondary data stream, and an input executor may execute control commands corresponding to the input so as to control a device or system.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/430,842 filed Jun. 4, 2019, which is a continuation of U.S. patentapplication Ser. No. 15/253,435, filed Aug. 31, 2016, which claims thebenefit of U.S. Provisional Application No. 62/212,100, filed Aug. 31,2015, the entire contents of which are incorporated herein.

FIELD OF THE INVENTION

The present disclosure relates generally to optimizing sensor data. Inparticular, methods for dynamically analyzing and switching between datastreams from a plurality of sensors to achieve optimal performance in atarget application are described.

DESCRIPTION OF RELATED ART

Most computer technology devices today are equipped with multiple visualsensors. Such sensors can include CMOS and CCD devices that aresensitive to visual light (e.g. RGB), thermal, infrared, or ultraviolet,and may be capable of depth sensing or range finding. These array ofsensors are useful for a variety of purposes, such as gesturerecognition or 3D visualization and/or mapping. However, each type ofsensor has particular limitations. For example, IR sensors may wash outeasily in bright light, while RGB sensors can, depending on size andquality, have difficulty providing a clean image in low lightconditions. The ability of a device to provide particular functionalitycan be dependent upon the sensors used.

Known methods of providing sensor data to applications may not beentirely satisfactory for the range of applications in which they areemployed. For example, existing systems typically utilize a singlesensor, and limit optimization of the data to techniques such as gainboosting and/or noise reduction. Such singular approaches may remainlimited by the fundamentals of the sensor technology employed.

Alternately, the use of multiple sensors to accomplish the same task mayimpose high demands for resources on a given system. For example, whilein principle it may be possible to fully process both an RGB data streamand a near infrared (NIR) data stream in parallel for some task such asgesture detection, the computational loads, power consumption, heatdissipation, etc. associated with processing two data streams end-to-endmay be undesirable for at least certain applications.

BRIEF SUMMARY OF THE INVENTION

The embodiments are directed to a variety of systems, apparatuses,methods, and paradigms for sensor switching.

In one embodiment of the present invention, a machine implemented methodis provided that includes disposing a processor in communication withsensors, each sensor being adapted to provide a primary data stream. Themethod includes instantiating a sensor controller and an operatingsystem on the processor. The method also includes instantiating a datastream analyzer on the processor, the data stream analyzer including adata stream standard. The method further includes instantiating an inputdetector, an input executor, and a data stream standard on theprocessor. The method includes communicating the primary data streamsfrom the sensors to the sensor controller, communicating the primarydata streams from the sensor controller to the operating system, andcommunicating the primary data streams from the operating system to thedata stream analyzer. The method also includes comparing the primarydata streams against the data stream standard with the data streamanalyzer, selecting with the data stream analyzer a secondary datastream most closely matching the data stream standard from among theprimary data streams, and communicating the secondary data stream fromthe data stream analyzer to the input detector. The method furtherincludes detecting with the input detector an input in the secondarydata stream, communicating the input from the input detector to theinput executor, and executing a control command with the input executorcorresponding with the input. The method also includes iterating:communicating the primary data streams from the sensors to the sensorcontroller to the operating system to the data stream analyzer,comparing the primary data streams against the data stream standard,selecting the secondary data stream, communicating the secondary datastream from the data stream analyzer to the input detector, detectingthe input in the secondary data stream, communicating the input from theinput detector to the input executor, and executing with the inputexecutor the control command corresponding with the input, wherein aprimary data stream selected as the secondary data stream in a firstiteration differs from a primary data stream selected as the secondarydata stream in a second iteration.

In another embodiment of the present invention, a machine implementedmethod is provided that includes disposing a processor in communicationwith sensors, each sensor being adapted to provide a primary datastream, and disposing the processor in communication with at least onedata stream recipient. The method includes establishing in the processora data stream standard, communicating the primary data streams from thesensors to the processor, comparing the primary data streams against thedata stream standard with the processor, and selecting with theprocessor a secondary data stream most closely matching the data streamstandard from among the primary data streams. The method furtherincludes communicating the secondary data stream from the processor tothe data stream recipient.

The method may include iterating comparing the primary data streamsagainst the data stream standard with the processor, and iteratingselecting with the processor the secondary data stream most closelymatching the data stream standard from among the primary data streams,wherein a primary data stream selected as the secondary data stream in afirst iteration differs from a primary data stream selected as thesecondary data stream in a second iteration.

The data stream standard may be specific to the data stream recipient.The data stream standard may be specific to a task for the data streamrecipient.

The data stream standard may include an environment factor. Theenvironment factor may include a minimum light level, a maximum lightlevel, a maximum infrared light level, and/or a maximum ultravioletlight level.

The data stream standard may include a content factor. The contentfactor may include a minimum dynamic range, a maximum noise, a minimuminput level, and/or a maximum input level.

The data stream standard may include a function factor. The functionfactor may include input detection, input confidence, input stepdetection, and/or input step confidence.

The sensors may include an RGB image sensor, a gray scale image sensor,a NIR image sensor, a TIR image sensor, a NIR time-of-flight depthsensor, an RGB stereo sensor, a grayscale stereo sensor, a NIR stereosensor, and/or an ultrasonic depth sensor.

Comparing the primary data streams may include assigning a quality valueto each of the primary data streams with respect to the data streamstandard, and selecting the secondary data stream includes selecting oneof the primary data streams exhibiting a best quality value. The qualityvalue may be at least two dimensional.

In another embodiment of the present invention, a machine implementedmethod is provided that includes disposing a processor in communicationwith sensors, each sensor being adapted to provide a primary datastream, and disposing the processor in communication with at least onedata stream recipient. The method includes establishing in the processora data stream standard, communicating the primary data streams from thesensors to the processor, comparing the primary data streams against thedata stream standard with the processor, and synthesizing with theprocessor a secondary data stream more closely matching the data streamstandard than the primary data stream from at least one of the primarydata streams. The method further includes communicating the secondarydata stream from the processor to the data stream recipient.

In another embodiment of the present invention, an apparatus is providedthat includes a processor and sensors in communication with theprocessor, each sensor being adapted to provide a primary data stream.The apparatus includes a sensor controller instantiated on the processorand adapted to control the sensors and adapted to receive the primarydata streams therefrom, an operating system instantiated on theprocessor and adapted to receive the primary data streams from thesensor controller, and a data stream analyzer instantiated on theprocessor and adapted to receive the primary data streams from theoperating system, the data stream analyzer including a data streamstandard and being further adapted to compare the primary data streamsagainst the data stream standard and to select a secondary data streammost closely matching the data stream standard from among the primarydata streams. The apparatus also includes an input detector instantiatedon the processor and adapted to receive the secondary data stream fromthe data stream analyzer and to identify an input in the secondary datastream, and an input executor instantiated on the processor and adaptedto receive the input from the input detector and to execute a controlcommand corresponding with the input.

In another embodiment of the present invention, an apparatus is providedthat includes a processor and sensors in communication with theprocessor, each the sensor being adapted to provide a primary datastream. The apparatus includes a data stream analyzer instantiated onthe processor and adapted to receive the primary data streams from thesensors, the data stream analyzer including a data stream standard andbeing further adapted to compare the primary data streams against thedata stream standard and to select a secondary data stream most closelymatching the data stream standard from among the primary data streams.The apparatus further includes a data stream recipient in communicationwith the processor and adapted to receive the secondary data stream fromthe data stream analyzer.

The data stream standard may include an environment factor, a contentfactor, and/or a function factor.

The sensors may include an RGB image sensor, a gray scale image sensor,a NIR image sensor, a TIR image sensor, a NIR time-of-flight depthsensor, an RGB stereo sensor, a grayscale stereo sensor, a NIR stereosensor, and/or an ultrasonic depth sensor.

The data stream recipient may include a gesture input detector adaptedto identify an input in the secondary data stream.

The apparatus may include an input executor adapted to receive the inputfrom the gesture input detector and to execute a control commandcorresponding with the input.

In another embodiment of the present invention, an apparatus is providedthat includes a processor and sensors in communication with theprocessor, each the sensor being adapted to provide a primary datastream. The apparatus includes a data stream analyzer instantiated onthe processor and adapted to receive the primary data streams from thesensors, the data stream analyzer including a data stream standard andbeing further adapted to compare the primary data streams against thedata stream standard and to synthesize a secondary data stream moreclosely matching the data stream standard than the primary data streamsfrom at least one of the primary data streams. The apparatus furtherincludes a data stream recipient in communication with the processor andadapted to receive the secondary data stream from the data streamanalyzer.

In another embodiment of the present invention, an apparatus is providedthat includes means for disposing a processor in communication with aplurality of sensors, each the sensor being adapted to provide a primarydata stream, means for disposing the processor in communication with atleast one data stream recipient, and means for establishing in theprocessor a data stream standard. The apparatus includes means forcommunicating the primary data streams from the sensors to theprocessor, means for comparing the primary data streams against the datastream standard with the processor, and means for selecting with theprocessor a secondary data stream most closely matching the data streamstandard from among the primary data streams. The apparatus furtherincludes means for communicating the secondary data stream from theprocessor to the data stream recipient.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Like reference numbers generally indicate corresponding elements in thefigures.

FIG. 1 shows a schematic view of an example of a programmable computingdevice.

FIG. 2 shows a schematic view of an example of a mobile electronicdevice.

FIG. 3 shows an example method for switching between sensors for optimalperformance, in flow chart form.

FIG. 4 shows an example method for switching between sensors for optimalperformance, with regard to example implementation with executableinstructions on a processor, in flow chart form.

FIG. 5 shows an example method for switching between sensors for optimalperformance, considering environment properties, in flow chart form.

FIG. 6 shows an example method for switching between sensors for optimalperformance, specifically considering light level as an environmentproperty with RGB and NIR sensors in a head mounted display, in flowchart form.

FIG. 7 shows an example method for switching between sensors for optimalperformance, considering content properties, in flow chart form.

FIG. 8 shows an example method for switching between sensors for optimalperformance, considering function properties, in flow chart form.

FIG. 9 shows an example method for switching between sensors for optimalperformance, including synthesizing a secondary data stream, in flowchart form.

FIG. 10 shows an example system for switching between sensors foroptimal performance, in block diagram form.

FIG. 11 shows another example system for switching between sensors foroptimal performance, in block diagram form.

FIG. 12 shows an example system for switching between sensors foroptimal performance including input detectors and input executors, inblock diagram form.

FIG. 13 shows an example system for switching between sensors foroptimal performance including a single input detector and multiple inputexecutors, in block diagram form.

FIG. 14 shows a simplified example system for switching between sensorsfor optimal performance, in block diagram form.

FIG. 15 shows another example method for switching between sensors foroptimal performance, with regard to example implementation using asystem similar to FIG. 10 through FIG. 14, in flow chart form.

FIG. 16 shows an example hardware device for switching between sensorsfor optimal performance, in the form of a head mounted display, inperspective view.

FIG. 17 shows a block diagram of a processing system that may implementcertain example operations as described.

DETAILED DESCRIPTION OF THE INVENTION

The disclosed methods and apparatuses for switching between sensors foroptimal performance will become better understood through review of thefollowing detailed description in conjunction with the figures. Thedetailed description and figures provide merely examples of the variousinventions described herein. Those skilled in the art will understandthat the disclosed examples may be varied, modified, and altered withoutdeparting from the scope of the inventions described herein. Manyvariations are contemplated for different applications and designconsiderations; however, for the sake of brevity, each and everycontemplated variation is not individually described in the followingdetailed description.

Throughout the following detailed description, a variety of methods andapparatuses for switching between sensors for optimal performanceexamples are provided. Related features in the examples may beidentical, similar, or dissimilar in different examples. For the sake ofbrevity, related features will not be redundantly explained in eachexample. Instead, the use of related feature names will cue the readerthat the feature with a related feature name may be similar to therelated feature in an example explained previously. Features specific toa given example will be described in that particular example. The readershould understand that a given feature need not be the same or similarto the specific portrayal of a related feature in any given figure orexample.

Various disclosed examples may be implemented using electronic circuitryconfigured to perform one or more functions. For example, with someembodiments of the invention, the disclosed examples may be implementedusing one or more application-specific integrated circuits (ASICs). Moretypically, however, components of various examples of the invention willbe implemented using a programmable computing device executing firmwareor software instructions, or by some combination of purpose-specificelectronic circuitry and firmware or software instructions executing ona programmable computing device.

Accordingly, FIG. 1 shows one illustrative example of a computer,computer 100, which can be used to implement various embodiments of theinvention. Computer 100 may be incorporated within a variety of devicesand systems, including but not limited to consumer electronic devicessuch as personal media players, cellular phones, smart phones, personaldata assistants, global positioning system devices, and the like.

As seen in this figure, computer 100 has a computing unit 102. Computingunit 102 typically includes a processing unit 104 and a system memory106. Processing unit 104 may be any type of processing device forexecuting software instructions, but will conventionally be amicroprocessor device. System memory 106 may include both a read-onlymemory (ROM) 106A and a random access memory (RAM) 106B. As will beappreciated by those of ordinary skill in the art, both read-only memory(ROM) 106A and random access memory (RAM) 106B may store softwareinstructions to be executed by processing unit 104.

Processing unit 104 and/or system memory 106 are connected, eitherdirectly or indirectly, through a bus 116 or alternate communicationstructure to one or more peripheral devices. For example, processingunit 104 and/or system memory 106 may be directly or indirectlyconnected to additional memory storage such as a storage device 110,e.g. a hard disk drive, a removable optical disk drive, a removablemagnetic disk drive, a flash memory card, etc. Processing unit 104and/or system memory 104 also may be directly or indirectly connected toone or more input devices 112, and one or more output devices 114. Inputdevices 112 may include, for example, a keyboard, touch screen, a remotecontrol pad, a pointing device (such as a mouse, touchpad, stylus,trackball, or joystick), a scanner, a camera or a microphone. Outputdevices 114 may include, for example, a monitor display, an integrateddisplay, television, printer, stereo, or speakers.

Still further, computing unit 104 may be directly or indirectlyconnected to one or more communication devices 108. For example, suchcommunication devices may include but are not limited to networkinterfaces for communicating with a network. Certain such networkinterfaces sometimes may be referred to as network adapters and/ornetwork interface cards (NIC). A network interface may translate dataand control signals from computing unit 104 into network messagesaccording to one or more communication protocols, such as theTransmission Control Protocol (TCP), the Internet Protocol (IP), and theUser Datagram Protocol (UDP). These protocols are well known in the art,and thus will not be discussed here in more detail. An interface orother communication device 108 may employ any suitable connectionagent(s) for connecting to a network, including, for example, a wirelesstransceiver, a power line adapter, a modem, and/or an Ethernetconnection.

It should be appreciated that, in addition to the input, output andstorage peripheral devices specifically listed above, the computingdevice may be connected to a variety of other peripheral devices,including some that may perform input, output, storage, and/orcommunication functions, or some combination thereof (even if otherdevices as noted with regard to elements 108 through 114 above also arepresent with similar functionality). For example, the computer 100 maybe connected to a digital music player, such as an IPOD® brand digitalmusic player. As known in the art, such digital music players may serveas an output device for a computer (e.g., outputting music from a soundfile or pictures from an image file), a storage device, etc.

Likewise, computer 100 may be connected to or otherwise include one ormore other peripheral devices, such as a telephone. The telephone maybe, for example, a wireless “smart phone,” such as those featuring theAndroid or iOS operating systems. As known in the art, this type oftelephone communicates through a wireless network using radio frequencytransmissions. In addition to simple communication functionality, asmart phone may also provide a user with one or more data managementfunctions, such as sending, receiving and viewing electronic messages(e.g., electronic mail messages, SMS text messages, etc.), recording orplaying back sound files, recording or playing back image files (e.g.,still picture or moving video image files), viewing and editing fileswith text (e.g., Microsoft Word or Excel files, or Adobe Acrobat files),etc. Because of the data management capability of this type oftelephone, a user may connect the telephone with computer 100 so thattheir data maintained may be synchronized.

Still other peripheral devices may be included with or otherwiseconnected to a computer 100 of the type illustrated in FIG. 1. In somecases, a peripheral device may be permanently or semi-permanentlyconnected to computing unit 102. For example, with many computers,computing unit 102, a storage device 110 in the form of a hard diskdrive and/or a removable optical disk drive and an output device 114 inthe form of a display are semi-permanently encased in a single housing.

Yet other peripheral devices may be removably connected to computer 100,in addition or instead. Computer 100 may include, for example, one ormore communication ports (which may in at least certain instancesfunction as and/or be considered as communication devices 108) throughwhich a peripheral device can be connected to computing unit 102 (eitherdirectly or indirectly through bus 116). These communication ports maythus include a parallel bus port or a serial bus port, such as a serialbus port using the Universal Serial Bus (USB) standard or the IEEE 1394High Speed Serial Bus standard (e.g., a Firewire port). Alternately oradditionally, computer 100 may include a wireless data “port,” such as aBluetooth interface, a Wi-Fi interface, an infrared data port, or thelike.

It should be appreciated that various embodiments of a computing devicemay include more components than computer 100 illustrated in FIG. 1,fewer components than computer 100, and/or a different combination ofcomponents than computer 100. Some implementations of the invention, forexample, may employ one or more computing devices that are intended tohave a very specific functionality, such as a digital music player orserver computer. These computing devices may thus omit certainperipherals or examples of peripherals shown in FIG. 1, such as anetwork interface, removable optical disk drive, printers, scanners,external hard drives, etc. Some implementations of the invention mayalternately or additionally employ computing devices that are adapted tobe capable of a wide variety of functions, such as a desktop or laptoppersonal computer. These computing devices may have substantially anycombination of peripheral devices or additional components, as desired.

In many examples, computers may define mobile electronic devices, suchas smartphones, tablet computers, or portable music players, oftenoperating the iOS, Symbian, Windows-based (including Windows Mobile andWindows 8), or Android operating systems.

With reference now to FIG. 2, an example mobile device, mobile device200, may include a processing unit 204 (e.g., CPU) configured to executeinstructions and to carry out operations associated with the mobiledevice. For example, using instructions retrieved from memory, theprocessing unit 204 may control the reception and manipulation of inputand output data between components of the mobile device. The processingunit 204 can be implemented on a single chip, multiple chips or multipleelectrical components. For example, various architectures can be usedfor the processing unit 204, including but not limited to dedicated orembedded processor, single purpose processors, controllers, ASIC, etc.By way of example, the controller may include microprocessors, DSP, A/Dconverters, D/A converters, compression, decompression, etc.

In certain cases, the processing unit 204 together with an operatingsystem may operate to execute computer code and produce and use data.The operating system may correspond to well-known operating systems suchas iOS, Symbian, Windows-based (including Windows Mobile and Windows 8),or Android operating systems, or alternatively to special purposeoperating system, such as those used for limited purpose appliance-typedevices. The operating system, other computer code and data may residewithin a system memory 206 that is operatively coupled to the processingunit 204. System memory 206 generally provides a place to storeexecutable instructions such as computer code and/or data that are usedby the mobile device. By way of example, system memory 206 may includeread-only memory (ROM) 206A, random-access memory (RAM) 206B, etc.Further, system memory 206 may retrieve data from storage devices 210,which may include for example a hard disk drive 210A, removable opticaldrive 210B, removable magnetic disk drive 210C, flash memory card 210D,solid state drive 210E, etc. Such storage devices may be fullyintegrated or partly/completely removable, for example hard disk drive210A that is integrated, an optical disc drive 210B that receives andplays removable DVDs, card slots for receiving media such as flashmemory cards 210D (or memory sticks), etc.

Mobile device 200 also may include input devices 212 operatively coupledto processing unit 204. Input devices 212 (when present) may beconfigured to transfer data from the outside world into mobile device200. As shown, input devices 212 may correspond to both data entrymechanisms and data capture mechanisms. In particular, input devices 212may include (but are not limited to) the following: force sensingdevices 212A such as force sensitive displays and housings; imagesensors 212B such as cameras; microphones 212C; touch sensing devices212D such as touch screens, touch pads and touch sensing surfaces;mechanical actuators 212E such as button or wheels or hold switches;motion sensing devices 212F such as accelerometers; and/or locationdetecting devices 212G such as global positioning satellite receivers,WiFi based location detection functionality, or cellular radio basedlocation detection functionality. Input devices 212 may also otherdevices and/or systems than those shown, e.g. a clickable displayactuator.

Mobile device 200 also may include various output devices 214, forexample being operatively coupled to processing unit 204. Output devices214 are configured to transfer data from mobile device 200 to theoutside world. Output devices 214 may include but are not limited to adisplay unit 214A as shown in FIG. 2 such as an LCD, LED, etc., and/ormay include other devices such as speakers or jacks, audio/tactilefeedback devices, light indicators, and the like.

Mobile device 200 also may include various communication devices 208, asmay be operatively coupled to the processing unit 204. Communicationdevices 208 may, for example, include an I/O connection 208A that may bewired or wirelessly connected to selected devices such as through IR,USB, or Firewire protocols, a radio receiver 208B which may beconfigured to communicate over wireless phone and data connections,and/or a global positioning satellite receiver 208C. Communicationdevices 208 may also include a network interface 208D configured tocommunicate with a computer network through various means which mayinclude but are not limited to wireless connectivity to a local wirelessnetwork, a wireless data connection to a cellular data network, a wiredconnection to a local or wide area computer network, or other suitablemeans for transmitting data over a computer network.

Mobile device 200 also may include a battery 218 as shown in FIG. 2.Additional related and/or comparable elements may include a chargingsystem, an AC power supply, some other power source, etc. For example,battery 218 may be charged through a transformer and power cord orthrough a host device or through a docking station. For example,considering a docking station as an example, charging may be transmittedthrough electrical ports, through an inductance charging means that doesnot require a physical electrical connection to be made, etc.

Various aspects, features, embodiments or implementations describedabove may be used alone or in various combinations. Methods may beimplemented by software, hardware, and/or a combination of hardware andsoftware. Embodiments also may be implemented as computer readable codeon a computer readable medium. The computer readable medium is any datastorage device that can store data that thereafter may be read by acomputer system, including but not limited to both transfer andnon-transfer devices as noted above. Examples of the computer readablemedia include but are not limited to read-only memory, random accessmemory, CD-ROMs, flash memory cards, DVDs, magnetic tape, optical datastorage devices, and carrier waves. The computer readable medium canalso be distributed over network-coupled computer systems so that thecomputer readable code is stored and executed in a distributed fashion.

With reference now to FIG. 3 an example method for switching betweensensors for optimal performance is now described. Method 300 functionsto supply at least one data stream from an optical sensor that isoptimized for such tasks as gesture recognition or 3-D mapping.Additionally or alternatively, method 300 can be used to provide asingle composite data stream, selected from the best portions of eachsensor data stream.

As used herein, the term “optimized” should be understood functionally,rather than to indicate a “best”, “perfect”, or “ideal” version. Withrespect to data streams, optimization of data streams refers herein toan arrangement for selecting a data stream (referred to as a secondarydata stream) from among multiple available data streams (referred to asprimary data streams), and/or for synthesizing a secondary data streambased on multiple available primary data streams. Typically, optimizingin such fashion may result in a superior outcome as opposed to using anyindividual primary data stream, e.g. by identifying primary datastream(s) that are well suited to whatever task is to be performed withthose data streams, and selecting one (or more) thereof as a secondary(“optimized”) data stream. For example, an “optimized” data stream willresult in the system functioning adequately, and/or in some sense moreeffectively, as opposed to the system's function without data streamoptimization.

However, it is emphasized that an optimized data stream may notnecessarily be “the absolute best” in objective or even subjectiveterms, nor is such required. For example, if two primary data streamsare approximately “just as good” (however defined), a determination ofwhich primary data stream is best is not necessarily required. Whiledetermining precisely which of those two primary data streams is “thebest” is not prohibited, in such instance it may be equally suitablesimply to choose either of the two primary data streams as a secondarydata stream, even if the primary data stream chosen is no better than(or is even worse than) the non-selected primary data stream.

Moreover, it is not required that an optimized data stream necessarilywill be either perfect or even particularly good, and again such is notrequired. For example, under certain lighting conditions a system withboth RGB and NIR sensing capabilities both RGB and NIR data streams maybe of poor quality for certain tasks; if there is no “good” data stream,then selecting either RGB or NIR as a secondary data stream (orsynthesizing a secondary data stream from the RGB and NIR data streamscombined) still may result in a poor quality secondary data stream.However, perfection is not required.

Thus, although it may be desirable for an optimized data stream to besuperior to all others and of excellent quality, in practice this maynot be the case, and is not required. While improved functionality overnon-optimized data streams is to be desired, it should be understoodthat 100% effectiveness is not necessary.

Again with reference to FIG. 3, method 300 provides an optimized datastream from an optical sensor. The arrangement in FIG. 3 may addresscertain challenges as may exist within conventional methods. Forexample, where a single sensor data stream is utilized, the particularproperties of that sensor may impact subsequent analysis, and in certaincircumstances, may inhibit desired functionality. A sensor that performswell in bright light, but that does not provide a clean signal inlow-light conditions, may limit performance of gesture recognition inlow light situations where that sensor is the only available source fora sensor data stream. Likewise, a sensor that addresses a limited visualspectrum, such as the near infrared (NIR), while omitting some or all ofthe visible and/or UV spectrum, may exhibit limited functionalitycompared with what might be enabled if additional data from UV and/orRGB sensors could also be considered.

As shown in FIG. 3, method 300 includes initializing multiple sensors ininitialization step 302. For purposes of example such sensors arereferred to at places herein as cameras and/or imagers, however this isan example only and should not be understood as limiting. Variousembodiments may use sensors other than cameras, imagers, etc., and theselection of sensors is not limited unless specifically noted.

Data streams are received in receiving step 316 from the varioussensors. Data streams from the sensors represent “raw materials” fromwhich an optimized data stream is selected and/or synthesized, thus theraw data streams received immediately from the sensors are referred toherein in certain instances as “primary data streams.”

The primary data streams from the sensors are analyzed and compared inanalysis and comparison step 322, with regard to the quality of thosedata streams, to determine one or more optimized data streams from amongthe primary data streams. Primary data streams may be compared againstone another, and/or may be compared against some external standard, andmay be compared in a variety of manners.

Still with reference to FIG. 3, the aforementioned one or more optimizeddata streams then are provided to one or more data stream recipients instep 326, such as a processor, an application or other data entityinstantiated on a processor, a hardware device, etc. Typically, thoughnot necessarily, such data stream recipients may use the optimized datastream(s) for further action and/or analysis. Optimized data streams arealso referred to herein as “secondary data streams”. As discussed abovewith reference to optimization, in some instances when a secondary datastream is selected from among primary data streams without modification,the secondary data stream may be identical or at least nearly identicalto one of the primary data streams. For clarity a distinction is madebetween primary data streams, which are raw data as provided by sensors,and secondary data streams, which are optimized data to be provided tosome “downstream” data stream recipient.

For clarity, as used in this application “processor” refers to acomputing system, which includes a processing unit and associated memoryand program storage, along with any necessary firmware and software toallow the processing unit to load an executable program from storage andexecute it. Similarly, to “instantiate” an application, data entity, orother executable entity encompasses the process of loading theexecutable application, data entity, program or other entity fromstorage into memory, and setting the processing unit to execute theentity.

Sensor initialization step 302 can be handled by a hardware controller(described further below), e.g. in conjunction with low-level softwareroutines, and initiates the various cameras and/or other sensors thatprovide primary data streams for analysis in step 322. Initializationmay include but is not limited to supplying power to the various camerasand/or other sensors, setting any adjustable parameters and features ofthe sensors, e.g. aperture, shutter speed, ISO/gain settings, aspectratio, spectrum, filters, frame rates, etc., and allocating hardwareresources such as memory buffers and interrupts. Followinginitialization step 302, in receiving step 316 the various primary datastreams from the cameras and/or other sensors are provided in a fashionsuitable for analysis. Some parameters, such as frame rates, forexample, may need to be synchronized in the initialization step,depending on the intended purpose of the primary data streams, thefunctional requirements of any recipient of a secondary data stream, theparticulars of the processor, etc.

In analysis and comparison step 322, the primary data streams from eachof the various cameras and/or sensors may be continuously monitored andanalyzed to determine suitability of the data stream for a predetermineddesignated purpose. Analysis and comparison step 322 may vary dependingon the designated purpose for which the secondary data stream is to beused. For example, where method 300 is being employed to supply asecondary data stream (or multiple secondary data streams) based onprimary data streams from an near infrared (NIR) sensor and an RGBsensor for gesture recognition, a clear signal for recognizing motionsmay be desired. Infrared sensors can wash out in bright light, such asfull sun, and may be poorly suited for gesture recognition in suchconditions. On the other hand, RGB sensors (e.g. conventional CMOSimagers) may function well in bright light, but may become indistinct inlow lighting situations, providing a mostly black image or an imageoverwhelmed with random noise where the gain of the sensor is maxed out.Where the lighting conditions in may vary widely, using a primary datastream from either a NIR or RGB sensor alone may be poorly suited forconsistent gesture recognition across all potential lighting conditions.For such an example application, analysis and comparison step 322 mayinclude monitoring and comparing the primary data streams from both theNIR sensor and the RGB sensor, and creating the secondary data stream byswitching between the two primary data streams depending on whether NIRor RGB is best suited for gesture recognition under the existingconditions at a given time.

Continuing to reference FIG. 3, if analysis and comparison step 322 ofprimary data streams determines that the NIR sensor is supplying aprimary data stream with predominately high-value pixels, which may beindicative of a washed out image, the RGB primary data stream may beused as the secondary data stream. Conversely, if the RGB primary datastream is predominately low or zero-value pixels, indicative of extremelow light, the NIR primary data stream may be used as the secondary datastream to provide a better result. It will be appreciated by a personskilled in the relevant art that this is merely one example of apossible analysis and, as mentioned above, the actual analysis beingperformed will vary depending on the intended use and purpose of thesensor data stream.

Moreover, analysis and comparison step 322 may be combined withinitialization step 302 and/or receiving step 316 to provide furtheroptions beyond simply switching between sensors. For example, dependingon the sensor package used in conjunction with method 300, it may besufficient to merely switch shutter speeds, apertures, and/or ISO orgain settings to improve performance, rather than switching to adifferent sensor (which may have less resolution, draw more power, haveother undesirable characteristics, etc.). In such a case, analysis andcomparison step 322 may, depending on implementation, loop back toinitialization step 302 to request a change in hardware status.

The results of analysis and comparison step 322 lead into providingoptimized data streams in step 326, where the optimized secondary datastream is provided to the application, functionality, or other datastream recipient that is to carry out some intended purpose. The formatof how the data stream is provided may vary depending on the needs andcapabilities of the data stream recipient. For example, a single datastream, comprised of frames dynamically switched between sensors, may beprovided where the application only needs a single stream, and benefitsfrom being “agnostic” as to the sensor source. Conversely, the datastreams from all sensors in the sensor package could be provided, alongwith a continual stream of instructions informing the receivingapplication of the desired or most optimal data stream, where theapplication can make use of multiple sensor data streams. Still further,providing an optimized secondary data stream in step 326 could combineideal data from any or all received streams to supply a single secondary(optimized) data stream that is a hybrid of some or all received primarydata streams (and/or that may include additional information). A personskilled in the relevant art will recognize that these are only examplesof possible output steps, and are not intended to be an exhaustive list.Providing an optimized secondary data stream in step 326 may be variedconsiderably so as to provide a secondary data stream or streams to thedata stream recipient.

As noted, numerous embodiments and variations of methods as describedherein are possible, and may be useful in at least certain instances.Several such variations are shown and described herein, though it shouldbe understood that these are examples only, and other arrangements maybe equally suitable.

Referring to FIG. 4, another example method 400 for switching betweensensors for optimal performance is shown. Although the arrangement inFIG. 4 may be similar at least conceptually to that in FIG. 3, method400 considers as an example an arrangement for implementation usingexecutable instructions instantiated on a processor. Such a processormay be present within an electronic device such as a smart phone, headmounted display, desktop PC, tablet, etc., although other arrangementsalso may be suitable.

In the method 400, a processor is disposed in step 402 in communicationwith two or more sensors. The sensors may vary considerably; forpurposes of explanation sensors are referred to as being cameras orother imagers, such as RGB cameras, NIR cameras, time-of-flight depthcameras, thermal cameras, etc. However, while such may be suitable,these are examples only, and are not limiting. Depending on theembodiment, disposing the processor in communication with sensors mayinclude physically connecting a processor to sensors, wirelesslyconnecting, initializing communication, making software or hardwareadjustments between sensors and processor, etc.

The method 400 also includes disposing the processor in communicationwith at least one data stream recipient in step 410. The data streamrecipient(s) may vary considerably; for purposes of explanation datastream recipients are referred to in places herein as applications,programming layers, or other data entities. However other arrangements,including but not limited to data recipients in the form of hardware,also may be suitable. As with disposing the processor to the sensors,disposing the processor in communication with the data stream recipientagain may include making a physical or wireless connection, initializinga device or data entity, etc.

A data stream standard is established in step 414 on the processor. Thedata stream standard represents some criterion by which a data streammay be evaluated (as described in more detail subsequently herein). Forexample, a data stream standard may represent a minimum or maximum lightlevel, or an intermediate range of light levels. Data stream standardsmay be simple, such as a “hard” cut-off of 10 lux as a minimum value;however, data stream standards also may be complex, including complexalgorithms, considerations of many dependent or independent factors,etc.

Furthermore, data stream standards may include some degree ofabstraction. For example, some numerical quality value, such as a “1through 10” rating, might be part of a given data stream standard, so asto represent the degree to which different levels of illumination aresuitable for a particular sensor. Use of such an abstract standard canallow for ad hoc comparison of different primary data streams that comefor sensors of varying technology. Where an abstract standard isemployed, primary data streams will typically be preliminarily examinedand their quality abstracted before being processed according to thedata stream standard. In one possible implementation, a primary datastream from an RGB sensor could be assigned the following scale on thebasis of measure light intensity: 5 lux or less might be rated as 1, 5to 7.5 lux as 2, 7.5 to 15 lux as 3, 15 to 25 lux as 4, etc. Conversely,where the primary data stream comes from a NIR sensor, the scale may berecalibrated for smaller fractions of lux. Such scales are preferablycalibrated so that each 1-10 rating approximately corresponds toequivalent primary data stream quality with due consideration given tosensor technology, viz. a “5” rating will yield approximately identicalquality regardless of sensor technology. Comparisons thus may beperformed regarding such ratings, rather than directly on the actuallight level. It should be appreciated that reference to light level isonly one possible example data stream standard, and that data streamstandards are not limited only thereto.

In addition, data stream standards may be specific to a given processor,device, sensor, and/or data stream recipient, and/or may vary based onthe settings or particulars of a processor, device, sensor, and/or datastream recipient. For example, a processor in communication with foursensors may have established four different data stream standards, onefor each sensor. In other embodiments, more than one data streamstandard per sensor or a single data stream standard for multiplesensors also may be suitable.

Establishing a data stream standard on the processor as shown in FIG. 4and elsewhere herein may encompass a range of events. For example, adata stream standard may be created computationally (e.g. in aprocessor), may be read from stored data (e.g. instantiated onto aprocessor from a hard drive or solid state drive), may be received ascommunication from some external system, defined based on inputs, somecombination thereof, etc. Unless otherwise specified, the manner bywhich various features may be established is not limited, nor is theactor (if any) that establishes those features limited.

Continuing in reference to FIG. 4, it is noted that data streamstandards (and also comparisons against data stream standards asdescribed in more detail below) may vary not only with regard toparticulars, but with regard to a general notion of what the standardaddresses. For example, at least three broad classes of data streamstandards may be considered: environment, content, and function. Theseshould not be understood as limiting.

An environment data stream standard may refer to some property of anenvironment. For example, continuing the previous discussions of lightlevel, minimum light level may be considered an environment property.Other environment properties may include, but are not limited to, lightlevels at specific wavelengths or bands of wavelengths (e.g.ultraviolet, 700 to 1100 nm, 489.3 nm sodium-D emission, etc.), averagelight levels over time, position in space, time, temperature, etc.Substantially any property that reflects some description of “the world”or some portion thereof may be considered as an environment property.

A content data stream standard may refer to some property of the datastream itself. For example, the dynamic range of an image frame, theminimum or maximum total input level for that frame, the minimum ormaximum input level of some channel for that frame (e.g. the highestred, green, or blue value for any pixel in a given image), the noiselevel within an image, the color balance within an image, the presence,size, persistence, etc. of “white out” or “black out” regions within animage, etc. may be considered content properties. Substantially anyproperty that reflects some description of the data stream itself orsome portion thereof may be considered as a content property.

A function data stream standard may refer to some property regarding thefunctionality and/or usability of a data stream for a given purpose. Forexample, if a camera feed is being considered for use in detectinggesture input, the ability to detect inputs, the ability to distinguishinputs from one another, the confidence in detecting inputs and/ordistinguishing inputs from one another, the ability or confidence todetect and/or distinguish individual steps in an input (e.g. particularstages of a complex gesture), etc. may be considered functionproperties. Substantially any property that reflects some description ofthe functionality of the data stream or some portion thereof may beconsidered as a function property.

It is noted that some degree of overlap may exist between environment,content, and function properties. For example, it may be possible todetermine the light level in an environment (an environment property) byevaluating the brightness of pixels in an image frame (a contentproperty). Thus an environment property (light level) may be determinedfrom the content of a data stream (camera feed). Such an arrangement isnot prohibited.

Furthermore, embodiments are not necessarily limited to considering onlyone of environment, content, and function properties. Data streamstandards (and comparisons thereof) may consider substantially anycombination of environment properties, content properties and functionproperties, and/or any other properties for which a data stream standardis defined. It is not required that all data streams in a givenembodiment consider the same properties, or that any data streamconsider the same properties under all conditions. Thus, data streamstandards may change over time due to changing conditions, differentcombinations of sensors, different data stream recipients, or for anyother appropriate reason that suggests an alteration to one or moreemployed data stream standards.

Continuing in FIG. 4, primary data streams are communicated in step 416from the sensors to the processor. Typically, though not necessarily,each sensor may deliver a single data stream. However, otherarrangements wherein two or more sensors cooperate to deliver oneprimary data stream (e.g. a stereo camera delivering two feeds forconsideration as a stereo pair) or wherein one sensor delivers two ormore primary data streams (such as a wide-spectrum camera deliveringnear infrared, visible, and ultraviolet bands as distinct data streams),also may be suitable.

As noted previously, a primary data stream is a data stream thatrepresents “raw material” for optimization. Typically, primary datastreams may be received directly from sensors, without modification, butthis is not required. For example, a video feed from a camera may bepre-processed in some fashion before being communicated in step 416 tothe processor although not strictly “raw data”, such a video feed stillmay be considered as a primary data stream herein.

In addition, it is not required that a primary data stream necessarilymust represent all data generated by a given sensor, or only datagenerated by that sensor. For example, if a camera defaults to producing60 frames per second but a particular task for which that data streammay be used requires updating only 10 times per second, frames may bediscarded (e.g. to reduce processing demands by considering fewerframes). Similarly, if a clock time or GPS position for the frames isconsidered useful, time or position stamps may be added to frames asmetadata. In either instance, the modified camera feed still may beconsidered as a primary data stream.

Continuing in FIG. 4, the primary data streams are compared in step 422against the relevant data stream standards in the processor. Thus, tocontinue the example of light levels, the light level of an RGB cameradata stream may be compared to some minimum light level.

Comparison may be independent or interdependent with regard to multipledata streams and data stream standards. That is, each primary datastream may be considered independently against a standard, to determine,for example, whether a given data stream is “good” or “good enough.”Such determinations can possibly be made through some numerical rating,though other approaches such as simple binary “yes/no” or “usable/notusable” evaluations also may be suitable. Alternatively, primary datastreams may be compared against one another in some manner; for example,it may be determined that a near infrared camera data stream is betterthan an RGB camera data stream at some point in time.

Comparisons may be simple or complex, and may be direct, such ascomparing dynamic range in one image feed vs. dynamic range in anotheror vs. some required value for dynamic range. Comparisons could also beindirect, such as determining quality values like the 1 to 10 scalediscussed above for the quality/usefulness of a data stream. Comparisonsmay consider multiple factors, for example dynamic range and resolution,or overall brightness and frame rate. In such instances standards and/orcomparisons may be “multi-dimensional”, that is, considering two or morefactors (x and y, or x, y, and z, etc.) rather than merely “higher isbetter” or similar.

Although data streams are referred to in places as being comparedagainst one another, and are described in places as being “good”, “goodenough”, etc., it should be understood that such evaluations may notnecessarily refer directly to the data itself. Considerations mayaddress environment and/or function in addition to or instead ofcontent. For example, if standards and comparisons address environmentproperties, such as light level, one data stream may be selected asbeing “optimal” based not on the actual content of that data stream, butinstead by the light levels where the sensor originating that datastream is located.

Still with reference to FIG. 4, a “best fit,” or optimized, data streamis selected in step 424 from among the primary data streams to createthe secondary data stream. The manner of selection is not limited, buttypically will depend on the particulars of the data stream standardsand the comparison of the data streams. For example, data streamstandards and comparisons considering light levels may be such that anRGB data stream is considered a best fit to standards at light levelsabove 40 lux, and that a NIR data stream is considered a best fit tostandards at light levels below 40 lux.

The simple “cut-off at 40 lux” approach described above is an exampleonly, and is not limiting. For example, determinations may becomputationally very complex, considering many different factors,weighted differently, and choosing from among many different primarydata streams, rather than just two such as RGB and NIR. In addition,determinations may not necessarily select one primary data stream asbeing distinctly the best fit. To continue the example above, for someembodiments an RGB data stream may be considered a best fit above 50lux, an NIR data stream would be a best fit below 25 lux, and the RGBand NIR data streams would be equally acceptable—in some sense both abest fit—between 25 and 50 lux.

Following selection, the selected primary data stream is considered thesecondary data stream, which may also be referred to as an optimizeddata stream. In step 426 the secondary data stream is communicated tothe data stream recipient. For example, a system implementing method 400that includes an RGB video feed, a NIR video feed, a stereo 3D sensorfeed and a time-of-flight depth sensor feed as primary data streams, anda gesture recognition application as a data stream recipient, wouldprovide an optimized secondary data stream selected from among theprimary data streams provided by the above-listed various feeds.Essentially, the method 400 picks out the best primary data stream interms of data for gesture recognition, and forwards that data stream tothe gesture recognition application, software layer, or other intendedrecipient.

Given such an arrangement, the data recipient need only consider dataalready evaluated as being optimized: the best available or at least asbeing good enough for intended purposes. It is not necessary for thedata recipient to devote resources to processing all four data streams;nor is it necessary for the data recipient to merely rely on a singledata stream in all conditions and evaluate possibly inferior data inhopes of still achieving acceptable performance. Thus, the arrangementin FIG. 4 may enable decreased resource demands (e.g. processor size,heat load, power consumption, processor cycles, etc.), and/or may enableimproved performance, such as by relying on RGB in well-lit environmentsbut using NIR in dimly-lit environments.

Method 400 does not explicitly illustrate any particular functions to beperformed with the secondary data stream, or the end results. Recipientsof the secondary data stream may perform a wide range of functions toobtain a variety of end results. For example, gesture recognition may beperformed by a device implementing method 400 upon the secondary datastream communicated in step 426 to the data recipient. Gestures sorecognized then might be provided to other applications or processesrunning upon the device as input, thereby controlling the processor,other applications or processes running upon the device, as well as anyother systems or hardware in communication with the processor. As afurther example of an application of method 400, the operation of a headmounted display or other electronic device might be controlled so as tochange what is shown on associated screens, open or close programs,change settings, send or receive messages, provide information orentertainment to a user of such a device, or any other task that thedevice is configured to perform.

Although step 426 refers to a single secondary data stream, otherembodiments may select and provide two or more secondary data streams.For example, if multiple data recipients are present with differentfunctions and/or data needs, each data recipient may receive a secondarydata stream optimized for that particular data recipient. Likewise, somedata recipients may require or benefit from multiple incoming streams ofdata. Furthermore, any given secondary data stream may be sent to morethan one data stream recipient, such as the same secondary data streambeing sent to both a gesture recognition application and a facerecognition program.

FIG. 5 shows an example method 500 for switching between sensors foroptimal performance, considering environment properties. As method 500in FIG. 5 shares a number of similar or identical steps to method 400,not all corresponding steps shown in FIG. 5 are described in detail. Thereader is referred to the foregoing discussion of method 400 for moredetail on those similar steps.

In method 500, in step 502 a processor is disposed in communication withsensors, and is further disposed in step 510 in communication with adata stream recipient.

A data stream standard is established in step 514 on the processor. Aspreviously noted, data stream standards (and comparisons thereto, etc.)may refer to environment properties; FIG. 5 shows an example arrangementof such. An environment property is established in step 514A. That is,in some manner—whether by human selection, algorithm, reading from adata store, etc.—some property of an environment is specified and madeavailable to the processor. In addition, an environment standard isestablished in step 514B for that environment property. Again, in somemanner (e.g. similarly) some minimum, maximum, range, numericalweighting, algorithm, etc. is specified and made available to theprocessor.

For example, the environment property and environment standard thereformay be disposed in a data storage device such as a magnetic hard drive,solid state drive, etc. When the processor is to perform steps in method500, the appropriate information may be instantiated onto the processorfrom the data storage device. In such an example steps 514A and 514B(and potentially the method as a whole) may in some sense be performedwhen some governing application is instantiated onto the processor,though it may also be suitable to consider such steps (and potentiallythe method as a whole) as being performed when that application iscoded, installed, or at some other point.

In addition, while establishing the environment property in step 514Aand establishing the environment standard in step S148 are shown asdistinct steps—for example, identifying what property is to beconsidered, and what particular values are considered relevant for thatproperty—this is an example only. In practice, it may not be necessaryor useful for all embodiments to establish a property as a distinct stepfrom establishing a standard, since in some sense “set a light level of25 lux as a minimum” at least implies that light level is the propertyunder consideration. Steps 514A and 514B are shown distinctly forclarity, but it should be understood that such steps may be combined incertain embodiments (and that other steps likewise may be combined,subdivided, reordered, etc. unless otherwise specified or required).

Moving on in FIG. 5, the primary data streams are communicated in step516 from the sensors to the processor.

The primary data streams are compared in step 522 to the data streamstandards in the processor. As noted with regard to step 514 above, inthe example method 500 of FIG. 5 the data stream standards refer to oneor more environment properties. Thus, comparison step 522 typically willaddress those environment properties.

Environment data is obtained in step 522A. That is, whatever aspects ofthe environment are considered with regard to the data stream standards,information on that aspect (or aspects) of the environment around thesensor, and typically though not necessarily around the processor aswell, is in some way acquired. For example, the environment data may beobtained in step 522A from additional sensors besides those for whichprimary data streams are evaluated. As a more concrete example,considering light level as an environment property, a light meter may bedisposed proximate the sensors which provide primary data streams, withthe light levels detected by that light meter then being obtained asenvironmental data. Alternately, environmental data may be obtained instep 522A from the data streams themselves. For example, insofar as anRGB camera detects images, typically that RGB camera operates bydetecting and measuring levels of light, notably in red, green, and bluebands. Thus, in at least some sense an RGB camera may be considered as,and may function as, a light meter, at least for the purposes of themethod 500. Other approaches also may be equally suitable.

The environment data is then compared in step 522B against theenvironment standard(s), on behalf of the primary data streams. In astrict sense the primary data streams themselves may not be comparedagainst either a standard or one another. However, as previously noted,such an arrangement is acceptable, and for purposes of simplicity maystill be referred to herein as “comparing the data streams”.Nevertheless, it may be worth noting that technically the comparison ofstep 522B may represent a comparison of information about the datastreams, rather than a comparison of the data streams themselves, in atleast certain embodiments. It should also be understood that if theenvironment data comes from the primary data streams themselves—e.g.light levels in images as in the example above—then the comparison ofstep 522B may be described even in very strict terms as a comparison ofthe data streams themselves.

A best fit of the primary data streams with regard to the environmentdata, environment standard, and comparison thereof is selected in step524 from among the primary data streams. Still with reference to FIG. 5,that secondary data stream is then communicated to the data streamrecipient in step 526.

Now with reference to FIG. 6, another method 600 is shown therein forswitching between sensors for optimal performance. The method 600 inFIG. 6 is presented as a specific and concrete example, addressing aparticular device (a head mounted display or “HMD”) with particularsensors (RGB and NIR cameras), a particular data recipient (a handrecognition layer), etc. Such concrete details as shown in FIG. 6 arepresented for clarity, however, it is emphasized that the method 600therein is an example only, and should not be considered as limiting.

In the method 600, RGB and NIR cameras that are hard-wired to aprocessor within an HMD are initialized in step 602. It should beunderstood that such initialization may represent a concrete example ofdisposing sensors in communication with a processor, as specific tocertain HMDs.

A gesture recognition layer is instantiated in step 610 on the processorof the HMD, being loaded from a solid state drive, also present withinthe HMD. Again, it should be understood that instantiating the gesturerecognition layer in step 610 may represent a concrete example ofdisposing a specific data recipient in communication with a processor.

In step 614, a data stream standard is established on the processor.This includes instantiating in step 614A the use of visible light levelsas an environment property onto the HMD processor from the solid statedrive, and also instantiating in step 614B a transition value of 30 luxfor visible light levels onto the HMD processor from the solid statedrive, as an environment standard.

The RGB and NIR camera streams (i.e. primary data streams) arecommunicated in step 616 from the RGB and NIR cameras respectively tothe HMD processor, via the hard-wired link between cameras and processor(as noted in step 602).

The RGB and NIR camera streams are compared in step 622 according to thedata stream standard within the HMD processor. This includes detectingin step 622A the visible light level for the local environmentsurrounding the RGB camera based on the data within the RGB camerastream itself. For example, images from the RGB camera may be evaluatedin the processor to determine, or at least approximate, the light levelin the region in the field of view of the RGB camera. Comparison step622 also includes comparing in step 622B the visible light level forthat environment to the 30 lux transition value in the HMD processor.

Still with reference to FIG. 6, in the HMD processor, the NIR camerastream is selected in step 624 as the secondary data stream for gesturerecognition if the visible light level in the environment of the RGBcamera is less than 30 lux. Otherwise, the RGB camera stream is selectedin step 624 as the secondary data stream.

The secondary data stream (in this instance either the RGB or NIR camerastreams, depending on visible light level) is communicated in step 626through the hard-wired link (referenced in step 610) to the gesturerecognition layer that is instantiated on the HMD processor. The gesturerecognition layer may be understood as at least similar to data streamreceivers as previously mentioned herein.

As may be observed, while the method 600 in FIG. 6 may in certain waysresemble that in FIG. 5, the method 600 in FIG. 6 also illustratesadditional steps as shown therein, showing an example through which asecondary data stream may be further utilized, e.g. for system controlthrough gesture input. However, this is an example only; in practice themethod may not necessarily specify to what use (if any) a secondary datastream may be put; thus certain steps as shown may be consideredoptional. Concrete use of a secondary data stream for system control isshown for illustrative purposes in the method 600 of FIG. 6, and shouldnot be considered limiting.

Continuing in FIG. 6, the method 600 includes identifying gesture inputin step 628 as may be present within the gesture recognition datastream. Such identifying step 628 is accomplished by/within the gesturerecognition layer. The input so identified is communicated in step 630to an application that also is instantiated on the HMD processor. Notethat the gestures themselves may or may not be communicated. Forexample, if a particular gesture is defined to represent an “openprogram” command, it may be suitable in at least certain instances tocommunicate an “open program” command from the gesture recognition layerto the application, rather than communicating the gesture. Applicationsthus may be “agnostic” as to what precise inputs are delivered.

The application then executes in step 632 whichever control command forthe HMD corresponds to the input communicated from the gesturerecognition layer. Thus, the HMD may change what is displayed, open orclose applications, alter settings, send or receive communications, etc.In at least some sense then the optimization of data streams as shown inFIG. 6 may facilitate control/operation of an HMD, or other device orsystem.

Now with reference to FIG. 7, an example method 700 for switchingbetween sensors for optimal performance, considering content properties,is described. Method 700 in FIG. 7 is at least somewhat similar to FIG.5 in concept, and not all corresponding steps shown in FIG. 7 aredescribed in detail.

In method 700, a processor is disposed in step 702 in communication withsensors, and the processor is disposed in step 710 in communication witha data stream recipient.

A data stream standard is established in step 714 on the processor. Inthe arrangement of FIG. 7, a content property is established in step714A. That is, some property of the content of one or more of theprimary data streams is identified. In addition, a content standard isestablished in step 714B for that content property: some minimum,maximum, range, numerical weighting, algorithm, etc. is specified andmade available to the processor.

While content properties and standards may differ from environmentproperties and standards in terms of what is being considered (e.g. thecontent of data streams rather than an environmental), it should beunderstood that comments made with regard to FIG. 5 as to how variousproperties and standards may be established, complexity thereof, etc.apply similarly to content properties and standards.

The primary data streams are communicated in step 716 from the sensorsto the processor.

The primary data streams are compared in step 722 to the data streamstandards in the processor. Since in example method 700 the data streamstandards refer to one or more content properties, comparison step 722typically will address content of data streams.

Content data is obtained in step 722A. That is, whatever features of thecontent are considered with regard to the data stream standards,information on that feature of the content is acquired, typically thoughnot necessarily through examination of the data streams themselves. Thecontent data is then compared in step 722B against the content standard.

A best fit of the primary data streams with regard to the content data,content standard, and comparison thereof is selected in step 724 fromamong the primary data streams. Still with reference to FIG. 7, thatsecondary data stream is then communicated in step 726 to the datastream recipient.

In FIG. 8 another example method 800 for switching between sensors foroptimal performance is depicted, in this instance, where the data streamstandard is directed to consider function properties. Again, wherecorresponding steps have been described previously herein, suchdiscussion may not be repeated in detail.

In method 800, a processor is disposed in step 802 in communication withsensors, and the processor is disposed in step 810 in communication witha data stream recipient.

A data stream standard is established in step 814 on the processor. Inthe arrangement of FIG. 8, a function property is established in step814A. That is, some property of the function of one or more of theprimary data streams is identified. In addition, a function standard isestablished in step 814B for that content property.

The primary data streams are communicated in step 816 from the sensorsto the processor.

The primary data streams are compared in step 822 to the data streamstandards in the processor. Because example method 800 addressesfunction properties, comparison step 822 typically will address contentof data streams.

Function data is obtained in step 822A. That is, whatever features ofthe function of the data streams are considered with regard to the datastream standards, information on that feature of the function isacquired. The function data is then compared against the contentstandard in step 822B.

Method 800 is not limited with regard to how the function data is to beobtained in step 822A. In certain embodiments, functional tests may beperformed on each primary data stream. Such an arrangement mightcolloquially be referred to as “pre-testing” the primary data streams.For example, if the intended data stream recipient is known to performgesture recognition, the primary data streams or portions thereof may beanalyzed to determine whether hands or gestures are present therein, thedegree to which hands or gestures are visible or distinct, theconfidence with which hands or gestures may be identified therein, etc.Such an approach may utilize either the same or different approachesthan the data stream recipient; using the same approach may provideconfidence that results of analysis in obtaining the data in step 822Amay closely represent the actual gesture recognition later on, whileusing a different approach may allow for faster and/or less resourceintensive analysis, such as through using simpler analysis tools,analyzing only part of an image frame, etc.

However, in other embodiments the actual functionality of data streamswithin the data stream recipient may be considered to determine functiondata, in addition to or in place of pre-testing analysis. For example,as shown in FIG. 8 method 800 may loop back to step 822 after asecondary data stream has been selected and utilized by a data streamrecipient. That is, an already-chosen secondary data stream may beevaluated as to how well that secondary data stream functions for thedata stream recipient, and the degree to which that secondary datastream recipient may influence further ongoing determinations as towhich of the primary data streams is selected as a secondary datastream. As a more concrete example, if an RGB camera feed has beenselected as a secondary data stream as being “optimum” (by whateverstandard), but in practice a gesture recognition program (data streamrecipient) cannot reliably distinguish one hand position from another,then the poor performance of the RGB camera feed may be considered infuture comparisons 822 of primary data streams. Such an approach may bereferred to as “post-testing” as distinct from the “pre-testing” notedabove.

Post-testing in such manner may exhibit advantages in at least certaininstances. For example, post-testing is not based on predictions of whatdata streams should be most useful for a particular purpose, but ratheron whether a given data stream is in fact useful, and possibly howuseful, for that purpose.

If, for example, unforeseen factors affect the performance of sensors ortheir data streams, it may still be possible through the method 800 toidentify an optimum data stream. More concretely, consider an RGB camerafeed that is of good quality by some standard, but that fails to producegood results. One possible approach may be if a gesture recognitionprogram relies on color models when evaluating data streams, and theuser is wearing gloves of some unusual color (e.g. lime green orelectric blue, colors not typical for skin tones). Even if the RGBcamera feed is clear, well lit, of good resolution, etc., the gesturerecognition program may be unable to discern gestures within that feed.Future comparisons of the RGB camera feed with (for example) atime-of-flight depth camera feed that includes color data may considerthe actual (poor) performance of the RGB camera feed.

Typically, though not necessarily, function post-testing may be used incombination with some other approach, such as function pre-testing,environment or content considerations, etc. This may be understood, inthat post-testing typically presumes that a primary data stream hasalready been selected as a secondary data stream and sent to a datastream recipient. However, while a combination with another approach maybe useful, such is not required. In some embodiments a secondary datastream may be selected from a designated default primary data stream(e.g. “always start with the RGB camera feed”), randomly from amongprimary data streams, or through some other approach that does notrequire immediate feedback from the data stream recipient.

In addition, the above example—an RGB camera feed with color data and atime-of-flight depth camera that does not include color data—illuminatesa point previously noted, namely, that different data streams may havedifferent properties and/or standards, and may be compared differently.For example, evaluation of an RGB camera feed may consider the colordistribution therein, while evaluation of a time-of-flight depth cameramay exclude consideration of color since no color data is provided, evenif both feeds are under consideration for the same gesture recognitionapplication as a data stream recipient. Thus, while similarity inproperties, standards, etc. for different sensors, different data streamrecipients, etc., is not prohibited, it may be understood thatvariation, including extreme variation, also is not prohibited, and maybe advantageous.

Likewise, as the criteria by which different primary data streams arejudged may be uneven among different data streams, the “fairness” ofdeterminations also may vary. Certain embodiments may be deliberatelyweighted or otherwise biased towards one sensor or primary data streamat the expense of others. For example, if it is known that a data streamrecipient processes time-of-flight depth camera data much moreefficiently than RGB camera data, methods may be weighted to preferselecting the time-of-flight depth camera data stream over the RGBcamera data stream. Thus, even if by some standard the RGB camera isproducing “better” data than the TOF depth camera, selection may favorthe TOF depth camera.

Returning to FIG. 8, regardless of particulars a best fit of the primarydata streams with regard to the function data, function standard, andcomparison thereof is selected in step 824 from among the primary datastreams. That secondary data stream is then communicated in step 826 tothe data stream recipient. As noted above, in certain embodiments instep 828 that data stream recipient may utilize the secondary datastream. The functionality of that secondary data stream may then bedetermined, and considered for future iterations of step 822 (asindicated by the arrow looping back from step 828 to step 822). However,while post-testing in such manner is acceptable and may be advantageous,post-testing is not required, and other arrangements also may besuitable.

Turning to FIG. 9, another example method 900 is shown for switchingbetween sensors for optimal performance. However, where in certainprevious examples herein a secondary data stream is selected from amongseveral primary data streams, in method 900 a secondary data stream isshown to be synthesized instead.

In method 900 a processor is disposed in step 902 in communication withsensors, and the processor is also disposed in step 910 in communicationwith a data stream recipient. A data stream standard is established instep 914 on the processor. Primary data streams are communicated in step916 from the sensors to the processor, and those primary data streamsare compared in step 922 to the data stream standards in the processor.FIG. 9 does not specify environment, content, or function comparisons;any combination thereof, or alternative arrangements, may be suitable.

Based on comparison step 922 of primary data streams to the data streamstandards, a secondary data stream is synthesized in step 924 in theprocessor utilizing one or more of the primary data streams.

The manner by which a secondary data stream is synthesized in step 924may vary considerably, and is not limited except as otherwise specified.Typically, though not necessarily, synthesis step 924 of the secondarydata stream includes combining certain features of one primary datastream with certain features of another primary data stream.Alternatively, combinations of more than two data streams also may besuitable. Considering gesture recognition as an example data streamrecipient, a high-resolution RGB camera signal may provide high qualitydata defining the edges of a hand and fingers thereof, while a TOF depthcamera may have less precise data on the outlines of the hand but couldprovide data as to whether fingers are extended towards and/or away fromthe camera position. A synthesized combination of a crisp outline withdepth data thus may be superior to either of the RGB or TOF feeds alone.

In colloquial terms, synthesizing primary data streams may produce asecondary data stream with “the best of both worlds”. Alternately,synthesizing a secondary data stream may allow weaknesses in individualprimary data streams to be de-emphasized or deleted, while retaininguseful data.

The number of possible primary data streams is extremely large, and thenumber of possible secondary data streams that could be usefullysynthesized therefrom even larger. Likewise, the number of ways thatprimary data streams could be combined to synthesize a secondary datastream is extremely large. Substantially any such combination may bepermissible, and embodiments are not limited with regard to what primarydata streams are combined in synthesis, or what sort of synthesis isperformed.

In addition, it is noted that synthesizing a secondary data stream fromonly one primary data stream also may be suitable for certainembodiments. For example, an RGB camera feed may be changed inresolution, color bit depth, frame rate, etc., visible features may beremoved or enhanced, edges may be sharped, etc.

As with secondary data streams selected from among primary data streams,secondary data streams that are synthesized may change over time, suchas with differing conditions. For example, a secondary data stream maybe synthesized from RGB and TOF data while light levels are high, butmay be synthesized from NIR and TOF data when light levels are low.Synthesized secondary data streams also may be weighted, utilizefeedback, etc. as previously described with regard to selected secondarydata streams.

Returning to FIG. 9, and regardless of the precise manner in which asecondary data stream is synthesized in step 924, that secondary datastream is then communicated in step 926 to the data stream recipient.

Now with reference to FIG. 10, an example system 1000 for switchingbetween sensors for optimal performance is shown, in block diagram form.System 1000 shown in FIG. 10 may be suitable for implementing themethods described herein, however this is not limiting. Methods notadapted to be implemented by system 1000 in FIG. 10 may be suitable, andsystems not adapted to implemented methods as shown elsewhere herein maybe equally suitable.

System 1000 includes a plurality of sensors 1004A through 1004D incommunication with a hardware control 1008, and providing primary datastreams 1008A through 1006D respectively thereto. The hardware control1008 is in turn accessed and controlled by an operating system 1010.Running on the operating system 1010 is software analysis 1012, whichimplements certain steps (e.g. as shown in method 300). Depending on howthe software analysis 1012 is implemented, one or more applications1016A through 1016C are adapted to receive an optimized secondary datastream 1014 that is provided by the software analysis 1012.

In the example shown in FIG. 10, the sensors 1004A through 1004D mayinclude camera types such as charge coupled device (CCD) orcomplementary metal oxide semiconductor (CMOS) sensors capable ofsensing luminosity levels (e.g. black and white pictures), RGB (e.g.color pictures), UV, IR, thermal, or other types of electromagneticradiation; depth sensors; 3D sensors; or any other type of sensor usefulfor analysis and detection appropriate to the intended application ofthe implementing system 1000 that is now known or later developed. Inthe particular arrangement shown in FIG. 10, four sensors 1004A through1004D are shown, namely an RGB sensor 1004A, a NIR sensor 10048, a 3Dsensor 1004C, and a depth sensor 1004D; however both the nature and thenumber of sensors 1004A through 1004D shown are examples only, and otherarrangements may be equally suitable.

The hardware control 1008 may be a hardware device, such as oneincluding supporting circuitry appropriate to the package of sensors1004A through 1004D, and may include power supplies andsignaling/control logic that provides a low level interface to thesensors 1004A through 1004D that is accessible by at a system softwarelevel. However, as noted below the use of a hardware-based hardwarecontrol 1008 is an example only, and other arrangements may be equallysuitable.

Still with reference to FIG. 10, the operating system 1010 is adapted tosupport general control of a system, e.g. communicating between varioushardware and software entities, etc. The operating system 1010 may be asoftware-based operating system instantiated on a physical processor,although other possible implementations, such as a firmware-basedoperating system or even a hard wired operating system could beutilized. The operating system 1010 entity may be any suitable operatingsystem for operating and controlling system 1000. The selection of anappropriate operating system 1010 may take into account the intendedpurpose of the system 1000, and its form factor. For mobile purposes,such as use in an appliance like Google Glass or a smartphone, amobile-oriented operating system such as Android, iOS, Windows Phone, orsimilar such operating system may be employed. Conversely, where system1000 is implemented in a more tethered or fixed application, such as alaptop or desktop computer, a desktop OS such as Windows, Mac OS Xavailable from Apple, Linux, or other UNIX variant may be an appropriateselection. Moreover, operating system 1010 may be custom-developed,purpose-specific, etc.

Software analysis 1012 is adapted to implement certain steps in methodsfor switching between sensors for optimal performance, including, butnot limited to, method 300. Software analysis 1012 is typically but notnecessarily implemented as part of a system library layer of softwarethat runs atop operating system 1010. When implemented as a systemlibrary software layer, software analysis 1012 may work internally tosupport the overall functionality of the system library. For example, asoftware analysis 1012 in the form of a software layer may beimplemented as part of a system library that offers gesture recognitionservices. Any or all of applications 1016A through 1016C (or other datarecipients) may tie into the system library to receive messages andinterpretations of recognized gestures. The system library, in turn, mayuse the software layer of to provide a secondary data streamselected/synthesized from the primary data streams from the package ofsensors 1004A through 1004D that is optimized for gesture recognition,and to which software analysis 1012 also performs gesture recognitiontechniques. When implemented in such fashion, the presence of softwareanalysis 1012 may be invisible to applications 1016A through 1016C and,rather than necessarily receiving an optimized sensor stream 1014,applications 1016A through 1016C may receive messages and/orinterpretations of recognized gestures, such as previously noted withregard to step 630 of FIG. 6. Other implementing system libraries mayinclude 3D mapping libraries that provide calling applications 1016Athrough 1016C with 3D information of the environment sensed by sensors1004A through 1004D. In such a use, software analysis 1012 may beconfigured to provide a secondary sensor stream 1014 that is optimizedfor the specific requirements of 3D mapping and sensing, which maydiffer from the requirements for gesture recognition.

Alternatively, software analysis 1012 may be implemented as a separatelibrary of its own, capable of being called by various applications,implemented as embedded functionality in a software-based operatingsystem 1010, implemented as hard-coded functionality in a hardware-basedhardware control 1008, etc., and may be software, microcode, orimplemented as a dedicated ASIC or hardware solution. Software analysis1012 also may be implemented as a component of any or all applications1016A through 1016C, where such functionality is not present or providedfor at a system or library level, thereby allowing implementation onsubstantially any system that is equipped with multiple sensors 1004Athrough 1004D.

Now with reference collectively to FIG. 11 through FIG. 13, severalvariations on systems for switching between sensors for optimalperformance are shown. As noted with regard to FIG. 10, apparatusesand/or other systems may vary considerably, for example with regard toelements being hardware or software, arrangements for communication,which elements carry out which functions, combining functions in oneelement and/or distributing functions among multiple elements, etc. FIG.11 through FIG. 13 show certain variations for purposes of illustration,however the examples shown are not limiting, and other arrangements alsomay be suitable.

Turning specifically to FIG. 11, therein another example system 1100 forswitching between sensors for optimal performance is shown, in blockdiagram form.

System 1100 includes a processor 1102 as shown, with certain otherentities (as noted below) instantiated as data and/or executableinstructions thereon. However, it is emphasized that such an arrangementis an example only, as may be seen by comparison with FIG. 10. In FIG.10 elements are not necessarily assumed to be software, and certainelements are considered (at least for purposes of example) as beinghardware, such as hardware control 1008. Generally speaking, elements ofa system for switching between sensors for optimal performance mayinclude elements in hardware form, as software, or as some combinationthereof; so long as the functions described may be carried out, theparticular form of elements or the system overall is not otherwiselimited.

Still with reference to FIG. 11, system 1100 includes sensors 1104Athrough 1104D. The sensors are not limited and may vary considerably. Asshown in FIG. 11 the sensors 1104A through 1104D are hardware devices incommunication with the processor 1102, though this is an example only.Also, where in FIG. 10 the sensors shown therein were specific sensorspresented as illustrative examples, sensors 1104A through 1104D in FIG.11 are referred to more generally. Thus, sensors 1104A through 1104D mayinclude cameras or other imagers in a variety of wavelengths (e.g.thermal infrared, near infrared, visible, ultraviolet, backscatterx-ray, etc.) and properties (e.g. monochrome, RGB or othermulti-channel, etc.), but may also include stereo imaging systems, depthimagers (such as time of flight cameras), ultrasonic sensors, motiondetectors, imaging spectrometers, millimeter radar, and so forth.

Sensors 1104A through 1104D are adapted to provide primary data streams1106A through 1106D respectively. As noted previously, it is notprohibited for one sensor to provide multiple primary data streams, orfor multiple sensors to cooperatively provide a primary data stream,though for clarity each of sensors 1104A through 1104D in FIG. 11provide one primary data stream 1106A through 1106D, so long as at leasttwo primary data streams are provided. In practice, certain embodimentsmay have only a single sensor, if that sensor is adapted to provide twoor more primary data streams. In addition, sensors not deliveringprimary data streams and/or providing data in addition to primary datastreams also may be present, for example for providing environmentproperty data, such as light meters, audio receivers, GPS receivers,etc. Such sensors are not shown in FIG. 11, but are not excluded fromother embodiments.

Primary data streams 1106A through 1106D are communicated to sensorcontrol entity 1108. Sensor control entity 1108 receives primary datastreams 1106A through 1106D, and may provide control of and/or supportto the sensors 1104A through 1104D, such as supporting drivers,coordinating resources (e.g. processor access), adjusting settings (e.g.for a camera focus, resolution, color bit depth, zoom, etc.), and soforth. It will be understood that sensor control entity 1108 may be toat least come degree analogous to the hardware control 1008 in FIG. 10.However, while the hardware control 1008 in FIG. 10 is described forsimplicity as a hardware device, and in the arrangement of FIG. 11 asensor control entity 1108 is shown as instantiated on the processor1102, analogous elements may be present as hardware, as software, and/oras some combination thereof (or in some other form). Moreover, althoughthe sensor control entity 1108 is shown as being separate from thesensors 1104A through 1104D and generic to all of the sensors 1104Athrough 1104D, a sensor control entity 1108 that is integrated with asensor, or that is specific to a sensor (e.g. with each sensor havingits own sensor control entity), also may be suitable. So long as thefunctions ascribed to sensor control entity 1108 may be carried out, theparticulars of sensor control entity 1108 are not otherwise limited.This variability of form applies likewise to other elements herein,unless otherwise specified.

Primary data streams 1106 are communicated to an operating system entity1110. While primary data streams 1106 are illustrated collectively as asingle arrow for simplicity, in practice data streams may becommunicated individually. Operating system entity 1110 typicallycontrols processor 1102 and/or other features within system 1100 and/orin communication therewith on at least an overall level, for examplecontrolling access to resources, providing a general user interface,supporting hardware, etc. Operating system entity 1110 is shown in FIG.11 as executable instructions and/or data instantiated on processor1102, though other arrangements may be suitable.

Typically, though not necessarily, operating system entity 1110 maycommunicate primary data streams 1106 with little or no alteration,rather acting as an intermediary between sensor control entity 1108 anddata stream analysis entity 1112 (see below), for communication of databetween two such potentially dissimilar and/or incompatible entities.However, it is not prohibited for an operating system entity 1110 to acton primary data streams 1106 in some fashion, whether simply bufferingdata for communication, recording data in a data store, communicatingdata to some external system, pre-processing primary data streams,adding metadata, etc.

Primary data streams 1106 are communicated from operating system entity1110 to a data stream analysis entity 1112. Data stream analysis entity1112 is adapted to perform certain functions related to optimizing datastreams as described already herein. For example, data stream analysisentity 1112 may include therein data regarding data stream standards,executable instructions for comparing primary data streams against datastream standards and/or one another, executable instructions forselecting or synthesizing a secondary data stream from the primary datastreams, etc. Thus data stream analysis entity 1112 is adapted toprovide a secondary data stream 1114. Broadly speaking, those functionsrelated to optimizing a secondary data stream 1114 from multiple primarydata streams as described herein in more detail with regard to FIG. 3through FIG. 9 (and also subsequent FIG. 14) may be performed and/oroverseen by data stream analysis entity 1112.

Typically, though not necessarily, data stream analysis entity 1112 maybe implemented as a software “layer” instantiated onto the sameprocessor 1102 as operating system entity 1110, and interfacingtherewith. Thus data stream analysis entity 1112 may in at least certainembodiments be instantiated as a “retrofit” on processor 1102 of anexisting device, interfacing with existing operating system entity 1110thereon and enabling advantages of data stream optimization as describedherein.

As a more concrete example, an existing smart phone with an existingoperating system, and with multiple sensors providing data, may acceptthereon a data stream analysis entity even if the smart phone were notalready designed or intended to perform data stream optimization asdescribed herein. In such an instance the data stream analysis entitymay be loaded onto the smart phone as a utility, an extension, etc.,providing additional functionality to existing systems withoutnecessarily requiring modification thereto.

Once data stream analysis entity 1112 has selected or synthesized asecondary data stream 1114 from primary data streams 1106 providedthereto, data stream analysis entity 1112 communicates secondary datastream 1114 to data stream recipients 1116A through 1116C for use bythose data stream recipients 1118A through 1116C. As shown in FIG. 11three data stream recipients 1116A through 1116C are shown, but more orfewer may be suitable in other embodiments. The nature of data streamrecipients 1116A through 1116C also is not limited; various applicationssuch as games, utilities, etc. that are interacted with directly by auser may serve as data stream recipients 1116A through 1116C, but otherentities such as gesture recognition, face recognition, security, etc.that are transparent to a user also may serve as data stream recipients1116A through 1116C. In addition, although sensor control entity 1108,operating system entity 1110, and data stream analysis entity 1112 areshown as distinct from data stream recipients 1116A through 1118C, anyor all of sensor control entity 1108, operating system entity 1110, anddata stream analysis entity 1112 may themselves also be data streamrecipients 1116A through 1116C. For example, secondary data stream 1114may be used for gesture input as a mechanism for controlling operatingsystem entity 1110, such as by changing system settings, opening orclosing other programs, and so forth. Likewise, although data streamrecipients 1116A through 1116C are shown in FIG. 11 as data and/orexecutable instructions instantiated on processor 1102, hardwaredevices, including but not limited to sensors 1104A through 1104D thatoriginate the primary data streams, also may be data stream recipients1116A through 1116C. The nature and number of data stream recipients1116A through 1116C is not limited, and may be extremely varied.

Moving on to FIG. 12, another example system 1200 for switching betweensensors for optimal performance is shown in block diagram form. At leastcertain features of the arrangement in FIG. 12 may be seen as resemblingFIG. 10 and/or FIG. 11, and detailed description thereof is notrepeated.

The system 1200 in FIG. 12 includes a processor 1202, with a sensorcontrol entity 1208, an operating system 1210, and a data streamanalysis entity 1212 instantiated thereon. The system 1200 also includessensors 1204A through 1204D in communication with the processor 1202,more particularly in communication with the sensor control entity 1208and adapted to communicate primary data streams 1206A through 1206Dtherewith. The primary data streams collectively 1206 are alsocommunicated from the sensor control entity 1208 to the operating systementity 1210, and from the operating system entity 1210 to the datastream analysis entity 1212.

However, as may be seen the communication paths between sensors 1204Athrough 1204D, sensor control entity 1208, operating system entity 1210,data stream analysis entity 1212 and elsewhere in FIG. 12 are shown astwo-directional arrows, where analogous communication paths in FIG. 10and FIG. 11 are shown as one-directional. Although typically key datamay move from sensors “up the chain” in one direction, as shown in FIG.12 communication may be two-way. Thus, just as sensor control entity1208 may receive communication from sensors 1204A through 1204D, such asprimary data streams 1206A through 1206D, sensor control entity 1208also may send communication to sensors 1204A through 1204D. Examples ofsuch communications may include changing settings of sensors, turningsensors on or off, or any other settings appropriate to the varioustechnologies implemented in sensors 1204A through 1204D. Likewise,communication between other elements in FIG. 12 and similar embodimentsmay be two-way, or may be more complex, such as each elementcommunicating with many elements in a network rather than simple one-wayor two-way communication paths.

In addition, as may be seen in FIG. 12 data stream analysis entity 1212provides three secondary data streams 1214A through 1214C, one to eachof three input detectors 1216A through 1216C. More broadly, inputdetectors 1216A through 1216C represent specific examples of a widevariety of possible data stream recipients. Thus in FIG. 12, each inputdetector 1216A through 1216C receives a different (or potentiallydifferent) secondary data stream 1214A through 1214C. As has been notedpreviously herein, different purposes may have different needs for data,and thus different secondary data streams 1214A through 1214C may eachbe differently optimized for different data stream recipients.Consequently, in at least certain embodiments a data stream analysisentity 1212 may select and/or synthesize two or more secondary datastreams 1214A through 1214C; one such secondary data stream may beoptimized for gesture detection, another for face detection, etc.

Also in FIG. 12, input detectors 1216A through 1216C themselves are incommunication with input executors 1220A through 1220C, communicatinginput 1218A through 1218C thereto. In the example arrangement of FIG.12, data recipients such as input detectors 1216A through 1216C are notnecessarily the end of the line. Rather, in the example as shown inputdetectors 1216A through 1216C each detect input present in an associatedsecondary data stream 1214A through 1214C, and then passes that input1218A through 1218C to a corresponding input executor 1220A through1220C. For example, each input detector 1216A through 1216C may detect adifferent type of input (e.g. gesture recognition, text recognition,face recognition, bar code/QR code recognition, etc.), and may thencommunicate that input 1218A through 1218C to input executors 1220Athrough 1220C.

Input executors 1220A through 1220C are adapted to execute some actionin response to input 1218A through 1218C communicated thereto. Forexample, input executors 1220A through 1220C may include applicationsthat respond to gestures by carrying out commands within thoseapplications in response to gestures that have been detected, anoperating system that similarly responds to gestures by carrying outcommands, a utility that responds to QR code input by translating theinput or forwarding to a web site, etc. Thus, through system 1200,control over system 1200 itself, or over some other system, may beenabled at least in part through optimizing data streams as shown anddescribed herein (e.g. by detecting input in optimized data streams andacting on such input).

Although FIG. 12 refers to call-outs 1216A through 1216C as inputdetectors, 1218A through 1218C as input, and 1220A through 1220C asinput executors, it is emphasized that addressing input is an exampleonly, and other arrangements may be suitable. That is, different datastream recipients other than or in addition to input detectors 1216Athrough 1216C may be present, and may perform functions other thandetecting input and passing along input 1218A through 1218C. Likewise,communications from data stream recipients may be other than input 1218Athrough 1218C. Similarly, entities receiving and/or acting on suchcommunications from data stream recipients (if present) also may beother than input executors 1220A through 1220C. The particular nature ortype of data stream recipients is not limited, nor communication by datastream recipients, nor any targets of such communication. Addressinginput, communicating that input, and executing that input as shown inFIG. 12 is presented as an example, and is not limiting.

Turning to FIG. 13, another example system 1300 for switching betweensensors for optimal performance is shown, in block diagram form. System1300 includes a processor 1302, with a sensor control entity 1308, anoperating system 1310, and a data stream analysis entity 1312instantiated thereon. System 1300 also includes sensors 1304A through1304D in communication with the processor 1302, more particularly incommunication with the sensor control entity 1308 and adapted tocommunicate primary data streams 1306A through 1306D therewith. Theprimary data streams, collectively 1306, are also communicated from thesensor control entity 1308 to the operating system entity 1310, and fromthe operating system entity 1310 to the data stream analysis entity1312.

Also in FIG. 13, a single data stream recipient 1316 is in communicationwith the data stream analysis entity 1312, and receives an optimizedsecondary data stream 1314. Data stream recipient 1316 in turncommunicates tertiary data 1318A and 1318B with tertiary recipients1320A through 1320C. Tertiary data 1318A and 1318B may be analogous withinput 1218A through 1218C in FIG. 12, and tertiary recipients 1320Athrough 1320C may be analogous with input executors 1220A through 1220Cin FIG. 12. However, as noted with regard to FIG. 12, embodiments arenot limited only to detecting, communicating, and executing input, and amore general arrangement thus is shown in FIG. 13 to illustrate this.

In addition, the arrangement in FIG. 13 includes only one data streamrecipient 1316, despite there being two tertiary data 1318A and 1318B,and three tertiary recipients 1320A through 1320C. As noted previouslyherein, various arrangements and numbers of data/data streams andentities communicating such may be suitable, and the numbers andarrangements thereof shown are examples only. FIG. 13 shows anarrangement wherein a single data stream recipient 1316 produces twotertiary data 1318A and 1318B, which in turn are communicated to threetertiary recipients 1320A through 1320C.

As a more concrete example, data stream recipient 1316 might represent agesture input detector receiving an optimized secondary data stream 1314from data stream analysis entity 1312 on processor 1302, detecting twodifferent types or classes of gesture input, and conveying the twodifferent inputs types as tertiary data 1318A and 1318B, respectively.For instance, certain gestures may be accepted only by certainapplications, or may mean different things for different applications;thus the same secondary data stream 1314 processed by a gesture inputdetector could yield two different sets of inputs. Continuing on,tertiary recipients 1320B and 1320C could be two applications thatreceive a common input of tertiary data 1318B, while another tertiaryrecipient 1320A is an application that requires and receives a differentinput of tertiary data 1318A, with each application acting on that inputaccording to the application's design.

The particular arrangements for systems shown in FIG. 10 through FIG. 13are examples only, and that many variations may be suitable. Inaddition, functions or elements shown separately for clarity may becombined, e.g. a data stream analysis entity may be integrated with anoperating system; likewise functions or elements may be subdivided,rearranged, etc., so long as the functions specified herein may becarried out.

Turning to FIG. 14, another example system 1400 for switching betweensensors for optimal performance is shown in block diagram form. System1400 includes a processor 1402, with a data stream analysis entity 1412instantiated thereon. System 1400 also includes sensors 1404A and 1404Bin communication with processor 1402, more particularly in communicationwith data stream analysis entity 1412 and adapted to communicate primarydata streams 1406A and 1406B therewith. A data stream recipient 1416 isin communication with the data stream analysis entity 1412,communicating a secondary data stream 1414 thereto. Secondary datastream 1414 will have been selected/synthesized from among the primarydata streams 1406A and 1406B by the data stream analysis entity 1412.

As may be understood through comparison of FIG. 14 with any of FIG. 10through FIG. 13, system 1400 in FIG. 14 may be considered a “bare bones”embodiment. That is, no separate sensor controller is present, andeither no operating system is present or at least is not directlyinvolved in carrying out the functions of the system 1400. Such anembodiment may be suitable for at least certain applications; as notedan operating system may function in certain embodiments as anintermediary for communicating data from one entity to another, and asensor controller likewise. Thus it may be possible and/or desirable incertain embodiments to dispense with a sensor controller, operatingsystem, etc., or at least to leave such elements “out of the loop” if noadditional functionality is provided by a sensor controller or operatingsystem in a particular embodiment.

However, it is noted that although certain embodiments may so exclude asensor controller, operating system, etc., excluding such elements isnot required, and the presence of such elements may in itself be useful.For example, certain existing devices may already include a sensorcontroller, have an operating system, etc., and embodiments wherein adata stream analysis entity is an add-on to existing systems may beuseful, such as by providing additional functionality in terms ofoptimal data streams to legacy devices that may already be in use. Thus,in at least certain instances a system for providing optimal datastreams may be established by instantiating a data stream analysisentity onto an already-existing device such as a smart phone, headmounted display, PC, etc.

Still with reference to FIG. 14, as may be seen the data streamrecipient 1416 is not shown as instantiated on the processor 1402, butrather as a separate element. As previously noted, any element may behardware, software, a combination of hardware and software, or someother configuration, so long as the functions as described herein may beperformed. Thus, as an example the data stream recipient 1416 may beconsidered as a hardware device, though the data stream recipient 1416could just as readily be executable instructions and/or datainstantiated on the processor 1402. Likewise, the data stream analysisentity 1412 could be hardware rather than being instantiated on theprocessor as shown, and the sensors 1404A and 1404B could instead beexecutable instructions and/or data on a processor 1402 that generates adata stream rather than hardware, etc.

Now with reference to FIG. 15, and also with regard to FIG. 10 throughFIG. 14, an example method 1500 is shown for switching between sensorsfor optimal performance, as may be implemented particularly with systemssimilar to those shown and described in FIG. 10 through FIG. 14.Although it is emphasized that methods as shown and described herein arenot limited only to implementation on such systems, it may beilluminating to show such a specific example method 1500 particularlysuited for such systems.

In the arrangement of FIG. 15, certain steps already described are notagain described in detail. Likewise references to steps carrying outcertain features/functions are not described in detail, where thosefeatures/functions themselves have been described previously herein.

In method 1500, a processor is disposed in step 1502 in communicationwith two or more sensors.

A sensor control entity is instantiated in step 1504 on the processor,an operating system entity is instantiated in step 1506 on theprocessor, and a data stream analysis entity is instantiated in step1508 on the processor. Sensor control entities, operating systementities, and data stream analysis entities and functions thereof havebeen described previously herein.

An input detector is instantiated on the processor in step 1510. Aspreviously noted, an input detector may be understood as an example of adata stream recipient. An input executor also is instantiated on theprocessor in step 1512. An input executor likewise may be understood asan example of a tertiary recipient.

A data stream standard is instantiated in step 1514 on the processor.Primary data streams are communicated in step 1516 from the sensors tothe sensor control entity.

The primary data streams are communicated in step 1518 from the sensorcontrol entity to the operating system entity, and are communicated instep 1520 from the operating system entity to the data stream analysisentity. Operating system entities and data stream analysis entities andfunctions thereof have been described previously herein.

The primary data streams are compared in step 1522 with the data streamstandard within the data stream analysis entity, and a best fit of theprimary data streams is selected in step 1524 as a secondary data streamwith the data stream analysis entity. The secondary data stream is thencommunicated in step 1526 from the data stream analysis entity to theinput detector. In step 1528 the input detector identifies input in thesecondary data stream, and communicates that input to an input executorin step 1530. The input executor then executes in step 1532 a controlcommand corresponding to the input.

As may be seen in FIG. 15, following step 1532 method 1500 loops back tostep 1516. Thus, method 1500 may be iterative and ongoing: primary datastreams are communicated in step 1516 from the sensors in an ongoingmanner, and result in the input executor executing in step 1532 controlcommands in an ongoing manner. In addition, it is pointed out that step1524 of selecting a secondary data stream also is ongoing in such case.Thus, over time different secondary data streams may be selected fromamong the primary data streams, for example as conditions changerelative to the data stream standard. In different iterations throughthe loop as shown in FIG. 15, different primary data streams may beselected in step 1524 for use as secondary data streams; the primarydata stream selected in one iteration may be different from the primarydata stream selected in a second iteration.

In summary, as circumstances change, method 1500 may select first onesensor feed as optimized, then another, and another, depending on whichof the various sensor feeds are well suited in the circumstances thatprevail at a given time. For example, an RGB camera feed may be selectedas an optimized data stream while a system is outdoors in bright light,but method 1500 may select a NIR camera feed as optimized if the systemis moved indoors into dimmer light, as night falls, etc.

Although FIG. 15 shows certain example features, such as selecting aprimary data stream in step 1524 as the secondary data stream, otherarrangements including but not limited to those described and shownelsewhere herein may be suitable. Thus embodiments similar to FIG. 15but wherein a secondary data stream is synthesized rather than selected,or where other variations exist, also may be suitable.

With regard again to the loop/update shown in FIG. 15, such updating maybe understood as potentially useful for devices that are mobile, forexample smart phones, head mounted displays, drones, vehicles, etc., asthose mobile devices move or are moved from one place with one set ofconditions to another place with another set of conditions. However,even substantially stationary devices, such as desktop PCs, gameconsoles, servers, etc. may in at least some instances benefittherefrom. For example, lighting in rooms may vary with time of day,whether lights are on and how many, etc. Different sensors may providedata of differing quality depending on many other factors including butnot limited to distance to targets of interest, rate of motion withinthe field of view, “busyness” in the field of view (e.g. how manypotential targets are present at a time), etc. Thus, circumstances alsomay vary for stationary systems, and adaptability by selecting amongdifferent data streams also may be useful.

While such ongoing updates as shown in method 1500 may not be necessaryfor all embodiments, ongoing updates may be suitable for at leastcertain embodiments.

Turning attention to FIG. 16, an example of a particular hardware device1600 that enables switching between sensors for optimal performance isdemonstrated. Hardware device 1600 includes a body 1620 and a sensorarray 1622. Body 1620 encloses a processing unit 1604 and a storagedevice 1610, as described above for example with respect to FIG. 1 andFIG. 2, such that the storage device 1610 may be in data communicationwith processing unit 1604 and include instructions executable by theprocessing unit 1604 so as to implement functions and/or methods forswitching between sensors for optimal performance as previouslydescribed and shown. Sensor array 1622 includes various visual sensors1612A through 1612D as disclosed previously herein, which may includebut are not limited to RGB, IR, UV, and 3D visual sensors. Moreparticular examples of such sensors may include 3D detection sensorssuch as Intel RealSense, Occipital Structure Sensor, Microsoft XBOXKinect sensors, etc. Such sensor packages may include a visual RGBsensor, IR depth sensors, and sometimes thermal sensors, such as thoseprovided by FLIR. In addition, useful embodiments may employ other thanvisual sensors: embodiments may be implemented with non-imaging and/ornon-visual sensors, such as sound, air pressure, chemical, RF, ionizingradiation, or other similar technologies. So long as sensors for aparticular sensed medium exist, with a range of possible sensors and/orsensor settings/configurations varying in capabilities, it is necessaryor desirable to provide an output from a collection of different sensorsthat is optimized for specific types of analysis, embodiments are notlimited with regard thereto.

Still with reference to FIG. 16, hardware device 1600 shown therein alsoincludes certain other features referred to previously herein. Acommunication device in the form of a radio receiver 1608 is shown, soas to enable communication from hardware device 1600 to other devices,systems, etc. In addition, hardware device 1600 as shown includesdisplays 1614A and 1614B, disposed on body 1620, so as to providegraphical output to a user as a possible function, such as stereographical output through the use of two such displays 1614A and 1614B.Such additional features, while not necessarily required for allembodiments, may be useful and are not prohibited.

It is noted that body 1620 shown for the example hardware device 1600 isillustrated with a form similar to a pair of glasses or a headset, andadapted to be worn in a similar fashion. Thus, hardware device 1600 maybe considered to be a head mounted display, although other arrangements,including but not limited to smart phones, tablets, desktop or laptopPCs, game consoles, drones, vehicles, etc. may be equally suitable.

Further, body 1620 as shown is configured with sensors 1612A through1612D positioned thereon such that when body 1620 is worn by a user,sensors 1612A through 1612D would be substantially aligned with thelines of sight of the user' eyes, and could potentially encompass fieldsof view at least somewhat comparable to those of the user's eyes,assuming sensors 1612A through 1612D exhibit fields of view similar inextent to those of the user. Similarly, in the arrangement shown body1620 is configured and displays 1614A and 1614B disposed thereon suchthat when body 1620 is worn by a user, displays 1614A and 1614B would beproximate to and substantially in front of the viewer's eyes. Such astereo configuration could enable stereo 3D graphical output to theuser. However, these arrangements are examples only.

Moving on to FIG. 17, therein is shown a block diagram of an apparatusthat may perform various operations, and store various informationgenerated and/or used by such operations, according to an embodiment ofthe disclosed technique. The apparatus may represent any computer orprocessing system described herein. Processing system 1700 is a hardwaredevice on which any of the other entities, components, or servicesdepicted in the examples of FIG. 1 through FIG. 16 (and any othercomponents described in this specification) may be implemented.Processing system 1700 includes one or more processors 1702 and memory1704 coupled to an interconnect 1706. Interconnect 1706 is shown in FIG.17 as an abstraction that represents any one or more separate physicalbuses, point to point connections, or both connected by appropriatebridges, adapters, or controllers. Interconnect 1706, therefore, mayinclude, for example, a system bus, a Peripheral Component Interconnect(PCI) bus or PCI-Express bus, a HyperTransport or industry standardarchitecture (ISA) bus, a small computer system interface (SCSI) bus, auniversal serial bus (USB), IIC (I2C) bus, or an Institute of Electricaland Electronics Engineers (IEEE) standard 1394 bus, also called“Firewire”.

Processor(s) 1702 is/are the central processing unit of the processingsystem 1700 and control the overall operation of processing system 1700.In some embodiments, processor(s) 1702 accomplish this by executingsoftware or firmware stored in memory 1704. Processor(s) 1702 may be, ormay include, one or more programmable general-purpose or special-purposemicroprocessors, digital signal processors (DSPs), programmablecontrollers, application specific integrated circuits (ASICs),programmable logic devices (PLDs), trusted platform modules (TPMs), orthe like, or a combination of such devices. A reader skilled in therelevant art will recognize that the term “processor” used throughoutthe foregoing is more analogous to processing system 1700 rather thanprocessor 1702, which forms only one component of the “processor”disclosed in the various methods and systems described in FIGS. 1through 15.

Memory 1704 is or includes the main memory of processing system 1702.Memory 1704 represents any form of random access memory (RAM), read-onlymemory (ROM), flash memory, or the like, or a combination of suchdevices. In use, memory 1704 may contain a code. In some embodiments,the code includes a general programming module configured to recognizethe general-purpose program received via the computer bus interface, andprepare the general-purpose program for execution at the processor. Inother embodiments, the general programming module may be implementedusing hardware circuitry such as ASICs, PLDs, or field-programmable gatearrays (FPGAs).

A network adapter 1712, a storage device(s) 1708, and I/O device(s)1710, are also connected to processor(s) 1702 through interconnect 1706.Network adapter 1706 provides processing system 1700 with the ability tocommunicate with remote devices over a network and may be, for example,an Ethernet adapter or Fibre Channel adapter. Alternatively or inaddition, network adapter 1706 can implement a wireless interface forcommunication with a corresponding wireless transceiver. Wirelessprotocols could include well-known standards such as WiFi, Bluetooth,NFC, any of the variety of cellular data communications protocolscurrently in use by commercial wireless providers, or any other similarwireless data communications technologies now known or later developed.Network adapter 1706 may also provide the processing system 1700 withthe ability to communicate with other computers within the cluster. Insome embodiments, processing system 1700 may use more than one networkadapter to deal with the communications within and outside of thecluster separately.

I/O device(s) 1710 can include, for example, a keyboard, a mouse orother pointing device, disk drives, printers, a scanner, and other inputand/or output devices, including a display device. I/O device(s) 1710also may include, for example, cameras and/or other imagers adapted toaccept visual input including but not limited to postures and/orgestures. The display device may include, for example, a cathode raytube (CRT), liquid crystal display (LCD), or some other applicable knownor convenient display device. The display device may take various forms,including but not limited to stereo displays suited for use in near-eyeapplications such as head mounted displays or other wearable devices.

The code stored in memory 1704 may be implemented as software and/orfirmware to program processor(s) 1702 to carry out actions describedherein. In certain embodiments, such software or firmware may beinitially provided to processing system 1700 by downloading from aremote system through processing system 1700, such as via networkadapter 1712.

The techniques herein may be implemented by, for example, programmablecircuitry such as one or more microprocessors programmed with softwareand/or firmware, or entirely in special-purpose hardwired,non-programmable circuitry, or in a combination of such forms.Special-purpose hardwired circuitry may be in the form of, for example,one or more AISCs, PLDs, FPGAs, etc.

Software or firmware for use in implementing the techniques introducedhere may be stored on a machine-readable storage medium and may beexecuted by one or more general-purpose or special-purpose programmablemicroprocessors. A “machine-readable storage medium”, as the term isused herein, includes any mechanism that can store information in a formaccessible by a machine.

A machine can also be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a laptop computer, a set-top box (STB), apersonal digital assistant (PDA), a cellular telephone, an iPhone, aBlackberry, a processor, a telephone, a web appliance, a network router,switch, or bridge, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine.

A machine-accessible storage medium or a storage device(s) 1708includes, for example, recordable/non-recordable media (e.g., ROM; RAM;magnetic disk storage media; optical storage media; flash memorydevices; etc.), etc., or any combination thereof. The storage mediumtypically may be non-transitory or include a non-transitory device. Inthis context, a non-transitory storage medium may include a device thatis tangible, meaning that the device has a concrete physical form,although the device may change its physical state. Thus, for example,non-transitory refers to a device remaining tangible despite this changein state.

The term “logic”, as used herein, may include, for example, programmablecircuitry programmed with specific software and/or firmware,special-purpose hardwired circuitry, or a combination thereof.

The disclosure above encompasses multiple distinct inventions withindependent utility. While each of these inventions has been disclosedin a particular form, the specific embodiments disclosed and illustratedabove are not to be considered in a limiting sense as numerousvariations are possible. The subject matter of the inventions includesall novel and nonobvious combinations and subcombinations of the variouselements, features, functions and/or properties disclosed above andinherent to those skilled in the art pertaining to such inventions.Where the disclosure or subsequently filed claims recite “a” element, “afirst” element, or any such equivalent term, the disclosure or claimsshould be understood to incorporate one or more such elements, neitherrequiring nor excluding two or more such elements.

Applicant(s) reserves the right to submit claims directed tocombinations and subcombinations of the disclosed inventions that arebelieved to be novel and nonobvious. Inventions embodied in othercombinations and subcombinations of features, functions, elements and/orproperties may be claimed through amendment of those claims orpresentation of new claims in the present application or in a relatedapplication. Such amended or new claims, whether they are directed tothe same invention or a different invention and whether they aredifferent, broader, narrower or equal in scope to the original claims,are to be considered within the subject matter of the inventionsdescribed herein.

The above specification, examples, and data provide a completedescription of the manufacture and use of the composition of theinvention. Since many embodiments of the invention can be made withoutdeparting from the spirit and scope of the invention, the inventionresides in the claims hereinafter appended.

The invention claimed is:
 1. A method, comprising: generating a firstdata stream; generating a second data stream independently from thefirst data stream; determining, by a data stream analyzer, a firstquality value for the first data stream; determining, by the data streamanalyzer, a second quality value for the second data stream; combiningone or more first portions of the first data stream with one or moresecond portions of the second data stream to synthesize a third datastream having a third quality value higher than the first quality valueand the second quality value, wherein the first quality value, secondquality value, and the third quality value are determined according to adata stream standard associated with a property of an environment; andcommunicating the third data stream from the data stream analyzer to aninput detector.
 2. A method, comprising: detecting, by a processingdevice, a first data stream; detecting, by the processing device, asecond data stream; synthesizing a third data stream by combining atleast a portion of the first data stream and at least a portion of thesecond data stream; determining, by the processing device, a firstquality value for the first data stream; determining, by the processingdevice, a second quality value for the second data stream; determining,by the processing device, a third quality value for the third datastream; selecting one of the first data stream, the second data stream,or the third data stream based on a highest relative value of the firstquality value, the second quality value, and the third quality value,wherein the selecting further comprises: iteratively comparing the firstdata stream and the second data stream against a data stream standardwith the processing device, and iteratively selecting the third datastream that more closely matches the data stream standard from among thefirst data stream or the second data stream; and communicating theselected one of the first data stream, the second data stream, and thethird data stream from the processing device to a data stream recipient.3. The method of claim 2, wherein the data stream standard is specificto the data stream recipient.
 4. The method of claim 2, wherein the datastream standard is specific to a task.
 5. The method of claim 2, whereinthe data stream standard comprises an environment factor correspondingto an environmental condition affecting the data stream recipient. 6.The method of claim 5, wherein the environment factor comprises at leastone of a light level, a position, a time, or a temperature conditionaffecting the data stream recipient.
 7. The method of claim 2, whereinthe first data stream corresponds with a first sensor or a first groupof sensors.
 8. The method of claim 7, wherein the second data streamcorresponds with a second sensor or a second group of sensors differentfrom the first sensor or the first group of sensors.
 9. The method ofclaim 2, wherein the first data stream and the second data stream areboth generated by a same sensor or a same group of sensors.
 10. Themethod of claim 9, wherein the first data stream and the second datastream are both generated by two or more sensors operatingcooperatively.
 11. The method of claim 2, wherein at least one of thefirst data stream or the second data stream is generated by at least oneof an RGB image sensor, a grayscale image sensor, a NIR image sensor, aTIR image sensor, a NIR time-of-flight depth sensor, an RGB stereosensor, a grayscale stereo sensor, a NIR stereo sensor, or an ultrasonicdepth sensor.
 12. The method of claim 2, wherein at least one of thefirst quality value, the second quality value, or the third qualityvalue is a multi-dimensional composite of at least two factors of thedata stream standard.
 13. The method of claim 12, wherein the highestrelative value is determined based on a multi-dimensional comparison ofthe first data stream, the second data stream, and the third data streamrelative to the at least two factors of the data stream standard.
 14. Amethod, comprising: determining, by a processing device, a first qualityvalue on a value rating scale for a first data stream based on a datastream standard associated with a property of an environment captured inthe first data stream; determining, by the processing device, a secondquality value on the value rating scale for a second data stream basedon the data stream standard associated with a property of an environmentcaptured in the second data stream; combining, by the processing device,the first data stream and the second data stream to synthesize a thirddata stream; determining, by the processing device, a third qualityvalue on a value rating scale for the third data stream; andcommunicating, to a data stream recipient, the third data stream whereinthe third quality value is greater than the first quality value and thesecond quality value.
 15. An apparatus, comprising: generating a firstdata stream; a data stream analyzer instantiated on a processing device,the data stream analyzer adapted to: receive a first data stream and asecond data stream; determine a first quality value for the first datastream, wherein a factor of the first quality value relates to at leastone first environmental property describing an environmental conditionat a source location of the first data stream; determine a secondquality value for the second data stream, wherein a factor of the secondquality value relates to at least one second environmental propertydescribing an environmental condition at a source location of the seconddata stream; combine the first data stream with the second data streamto synthesize a third data stream; determine a third quality value forthe third data stream, wherein the first quality value, the secondquality value, and the third quality value are values defined by a valuerating scale; an input detector instantiated on the processing device,the input detector being adapted to, in response to the third qualityvalue being greater than the first quality value and the second qualityvalue: receive the third data stream from the data stream analyzer; andto identify an input in the third data stream for sending; and an inputexecutor, instantiated on the processing device, the input executorbeing adapted to: receive the input from the input detector; and executea command corresponding to the input.
 16. An apparatus, comprising: aprocessing device; a first sensor and a second sensor both incommunication with the processing device, the first sensor being adaptedto provide a first data stream and the second sensor being adapted toprovide a second data stream; a data stream recipient in communicationwith the processing device, the data stream recipient being adapted toreceive a third data stream in response to a quality of the third datastream being greater than a corresponding quality of the first datastream and a corresponding quality of the second data stream, whereinthe third data stream is selected by: iteratively comparing the firstdata stream and the second data stream against a data stream standardwith the processing device, and iteratively selecting the third datastream that more closely matches a data stream standard from among thefirst data stream or the second data stream.
 17. The apparatus of claim16, wherein the data stream recipient comprises a gesture input detectoradapted to identify an input from the third data stream.
 18. Theapparatus of claim 17, further comprising an input executor adapted toreceive the input from the gesture input detector and to execute acorresponding command.
 19. An apparatus, comprising: a data streamanalyzer instantiated on a processing device, the data stream analyzerbeing adapted to: receive a first data stream and a second data stream;combine the first data stream with the second data stream to synthesizea third data stream; and a data stream recipient adapted to receive thethird data stream from the data stream analyzer in response to the thirddata stream having a quality value greater than a quality value of thefirst data stream and a quality value of the second data stream, whereinthe quality values are determined relative to a data stream standardhaving at least one factor corresponding to an environment of theapparatus.